What do you think of this concept? [Archive]

Blade556

05-16-2005, 05:29 PM

http://www.freewebs.com/blade556/bre.txt

This is something I've been toying around with for a while, but now I want to put it into production. There's still a few kinks I want to work out still, but since this is a rich ROM hacking community, I figure you would be able to help me out here and tell me if it's a novel idea or not.

I need a C++ coder (I may ask Lenophis, since I know he knows C++), but I might pay for a professional to code the jist of it.

And if you're wondering the ever asking "WHY?!", just read the whole thing and it should explain itself.

But more importantly, who would be willing to work on this with me? I think it's a grand idea, but just don't go about saying "Man, you have no idea what you're getting yourself into" or stuff like that. I fully realize how much work is involved with this. More than I can handle by myself, thats for sure. Which is why I need your help.

So tell me all your problems with it (besides what I said above), and we can sort it out here (Or IRC...I'm usually on Espernet, in #rom-hacking late at night).

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

ToneE040

05-16-2005, 07:09 PM

Splendid idea! It would definately be a lot of work, it would be nice to have that for Linux as well. This would definately be a big project. I'm impressed that you'd even pay people to help in the production. Best of luck, there's always room for improvement. <img src=smilies/thumb.gif>

Fx3

05-16-2005, 07:32 PM

Unless you're a genious, that's impossible. At last, what Kent 'SnowBro' did was something VERY similar with mapper #0 games - instead of pure ASM output, he rebuilt the code somewhat. For mappers, the task is much more complicated and... unless you know the meanings of EVERY peculiar aspect of the game, I (honesly) dunno...

CROcOp

05-16-2005, 10:33 PM

It is very possible, but very hard. It will probably take years to get this all done. When making a script inserter, make it change the pointers and automatically look for free space in rom. That is the tool i am dreaming of.

---
http://img.photobucket.com/albums/v736/CROcOp/crosig.jpg

Disch

05-16-2005, 10:49 PM

One thing I just noticed:

Why are you so attached to Word files? Is the .doc format even open for public use? Why can't you just use .txt files?

I don't even have Word installed on my computer -- I'm sure I'm not alone.

phonymike

05-17-2005, 04:23 AM

sorry not gonna happen. all those features would need to be programmed based on each individual game. it's only possible making custom programs to extract/insert scripts, graphics, sounds and the like on a per game basis. sure some companies reuse data storage techniques on several games, but no one program can do what you imagine.

immortality (http://www.alexchiu.com/affiliates/clickthru.cgi?id=phonymike)

hcs

05-17-2005, 06:51 AM

>Once it has finished searching the ROM for Assembly code, it will log all the different calls
>the game sends out to the processor in alphabetical order.

Why would you ever want this?

A disassembler that divides things into procedures/functions/subroutines//"processes" automagically would be awesome. I remember running through Arkanoid and wiritng down each fcn by hand, it was very useful tedious and seems like it would lend itself easily to automation, but then again I've never even tried.

-http://www.halleyscometsoftware.com/hcs</a>

Audigy

05-17-2005, 02:06 PM

"I have spent too
much time trying to figure out DOS commands and parameters and mumbo jumbo bullshit like
that. "

<img src=smilies/cwm27.gif><img src=smilies/cwm27.gif><img src=smilies/cwm27.gif><img src=smilies/cwm27.gif><img src=smilies/cwm27.gif>

--

http://www.oddigy.com
beadsprites and PSFs, oh my!

Blade556

05-17-2005, 02:28 PM

For some reason, Windows 98 won't display any Japanese characters except for the internet and in Word. I thought it would be great for translation, but perhaps just using text files will be fine.

...Do they show up in the XP edition?

And yes, DOS is a pain in the ass to learn. I just want a windows binary that does what I want it to do without the hassle. Naga, Atlas, Tile Molester...They don't do what I want and the readme doesn't tell you anything either.

Anyway, I hope I can get this to work. I know alot of the stuff cannot be automated, but we'll see about that soon enough...At least, I know the Script can't really be, which is why the program will need the start address for it.

And hcs, a game sends Mnemonic calls to the processor through the leads on the loading bay thing. The ROM sends out the signals too. Enigma will emulate the ROM (Not like an emulator, just to initialize it and run it minimally) and log those mnemonic signals. For logging the different game processes, they will probably be recorded by appearance.

But that would be tricky to do...I'm thinking of 2 screenshots per event, one from the very start of the process and to the end of it...But even then it's still gonna be a pain to code...

Oh well...I'll get to that problem as it comes.

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

hcs

05-17-2005, 03:28 PM

> And hcs, a game sends Mnemonic calls to the processor
> through the leads on the loading bay thing...

Er, yeah, I know what mnemonics (more specifically, instructions) are, I just don't seee why a count of them sorted alphabetically would be.
Your best hope of getting a program like this to work would in fact be to write an emulator, you could make attempts to locate the graphics in the ROM by tracing where the stuff actually accessed by the graphics hardware, same for sound, same for code for that matter. Ditto also for data, anything read but not "traced" to sound, graphics, or CPU could be considered additional data, given the instruction set you might be able to recognize arrays when series of data are accessed by indexed instructions.
Think of this, you run the program, load a ROM, and when some text comes on screen you select a tile and it'll tell you
a) where the graphics data itself came from (and you could continue to trace back to see where that originated, all the way back to the ROM)
b) what code wrote the command to put that specific tile up, and a stack trace back from there (which might help you find text routines and, with data references noted properly, the script you seek as well).
It'd need to track and obscene amount of data, the heuristics would naturally not cover every situation, but properly implemented it might be a great help.

> For logging the different game
> processes, they will probably be recorded by appearance.
> But that would be tricky to do...I'm thinking of 2
> screenshots per event, one from the very start of the
> process and to the end of it...

I assume that you here mean save states? I'm not too clear on what you mean by a "process", either. Tracking stack operations would probably be the best way to recognize what I think you mean

> Oh well...I'll get to that problem as it comes.

This isn't the sort of situation where you can write a program with a description, unless you have a good idea how it'll be implemented you're not asking for a C++ programmer, you're asking for someone to do the whole thing for you. At least have some idea.

-http://www.halleyscometsoftware.com/hcs</a>

Blade556

05-17-2005, 03:35 PM

> Er, yeah, I know what mnemonics (more specifically,
> instructions) are, I just don't seee why a count of them
> sorted alphabetically would be.

Yeah, I guess there is no point, looking back, but I figured (At the time) it might be a good idea...Anyway, just an option I'm leaving on the table.

> I assume that you here mean save states? I'm not too clear
> on what you mean by a "process", either. Tracking stack
> operations would probably be the best way to recognize what
> I think you mean

The initialization is a process. Fading in and out of 2 different screens is a process. In an RPG type game, leaving the city and going to the world map is a process. Every time the processor recieves a big change in Assembly signals is a process. I don't really know how to explain it, I just know what it is...I'm sure you've had that before :p I'll try later on to figure it out though...Sorry

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

Disch

05-17-2005, 05:42 PM

> For some reason, Windows 98 won't display any Japanese
> characters except for the internet and in Word. I thought
> it would be great for translation, but perhaps just using
> text files will be fine.

I doubt many translators use Word for anything translation related. People that have the need to view japanese characters have the means to do so (like with NJStar or something). Like I said not everyone has Word -- but everyone has Notepad or some other text editing program. Besides you can always open .txt files in Word if that's really what you want to do... but you can't really open .doc files in anything else (except for WordPad I guess), so going with .doc would be a big mistake imo.

> And yes, DOS is a pain in the ass to learn.

Command line driven != DOS -- don't confuse the two.

> Enigma will emulate the ROM (Not like an emulator, just to initialize
> it and run it minimally) and log those mnemonic signals. For logging the
> different game processes, they will probably be recorded by appearance.

You can only trace the code so far until it starts polling for user input (or some other system status) and you'll hit a wall. I can't really explain the problem that well, but you'll see what I'm talking about when you try to do this. There's more to emulating the system than just the CPU -- and those other aspects of the system can greatly affect the CPU's execution, making straight code tracing difficult (player input, PPU/GPU status, system/mapper/other hardware IRQs -- even the sound system in extreme cases [*cough* Mig-29 Soviet Fighter])

MooglyGuy

05-17-2005, 05:57 PM

> Every time the processor recieves a big change in Assembly
> signals is a process.

"a big change in Assembly signals"? What the fuck are you talking about? Do you even know what the fuck you're talking about, or are you just spewing technical-sounding bullshit to try to get people to think you have a clue (which you obviously don't)? <img src=smilies/headshake.gif>

CROcOp

05-17-2005, 06:53 PM

> "a big change in Assembly signals"? What the fuck are you
> talking about? Do you even know what the fuck you're
> talking about, or are you just spewing technical-sounding
> bullshit to try to get people to think you have a clue
> (which you obviously don't)?
>

Chill man, just chill. Enough of fucks, okay. Gee, all the guys here have weak nerves or some frustrations(not talking about you). So, let the guy talk. Eventually he will realize that the idea he just got isn't very much possible, and he will give up. Blade, if the thing you imagined is possible someone would probably make it by now.

---
http://img.photobucket.com/albums/v736/CROcOp/crosig.jpg

Ugly Joe

05-17-2005, 07:18 PM

I didn't read the whole document, so keep that in mind.

It seems like you want to make an all-encompassing rom hacking utility. From the rom hacking I've done (NES and some GBA), this seems near impossible, since each game handles graphics/sounds/whatever in a somewhat different manner. There are obviously similarities, but they seem to be too low level to be useful (for example, I can find in any NES rom where the game is drawing a tile to the screen and where in memory it's pulling the data from, but finding what put that data into that memory location and finding what process decided exactly what data to put into that location are far more difficult).

I think a driver-based tool would make more sense. Like, having a file specify where a pointer table and values are located in the rom, or where the graphic data is located/how it is stored. Basically, have all of the game-specific problems be handled outside of Enigma and let Enigma handle the actual rom editing once the dirty-work has been taken care of.

Blade556

05-17-2005, 07:30 PM

> I doubt many translators use Word for anything translation
> related. People that have the need to view japanese
> characters have the means to do so (like with NJStar or
> something).

I'll figure out what I'm going to do about dumping Japanese characters as soon as I can't think of anything else to add support for. I've never heard of NJstar, though...

> Command line driven != DOS -- don't confuse the two.

Is there a difference? Perhaps there is, but it doesn't matter. I'm trying to avoid anything DOS-like or command line related with this program.

> You can only trace the code so far until it starts polling
> for user input (or some other system status) and you'll hit
> a wall. I can't really explain the problem that well, but
> you'll see what I'm talking about when you try to do this.

> [*cough* Mig-29 Soviet Fighter])

...A mig fighter?

Enigma can send input signals automatically, for both yes, no, or more input choices that appear for anything.

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

Blade556

05-17-2005, 07:34 PM

That is a grand idea, and I think you've inspired me to do both. But then the drivers have to be made for each different game, until he have hundreds of drivers going around.

In any case, it's a great idea. Definitely something I'm going to put into alot of thought.<img src=smilies/thumb.gif>

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

Blade556

05-17-2005, 07:35 PM

http://filebox.vt.edu/users/rogrubb3/SNESDev/CodeLib/SNESINIT.ASM.txt This is what I meant</a>

Sorry for the confusion. <img src=smilies/upeyes.gif>

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

Disch

05-17-2005, 09:50 PM

> ...A mig fighter?

It's an NES game. One of the more difficult ones to emulate properly due to its heavy reliance on proper DMC behavior (an area of the NES's sound).

> Enigma can send input signals automatically, for both yes,
> no, or more input choices that appear for anything.

There's an infinite amount of input button presses that trigger different areas of code -- when tripped at different times, and in different orders. Everything in the game flow revolves around which buttons are pressed and when (random number generators are usually driven by user input as well).

A good way to imagine automating this process would be to put in a cartridge in your NES/SNES/whatever and just start hitting random buttons. You won't get very far with it -- and you certainly won't expose most of the code in the game. If you have a hundred computers working together and running around the clock punching in every sequence of player input imaginable it will still take years to expose all the code/data in a game (talking about smaller games -- when you get to larger more complex games it's even worse).

Imagine having a computer try and play through Megaman X or something -- having to press all the right buttons at all the right times (keep in mind the computer has no idea which buttons are right or wrong -- the outcome of the button presses is pretty much only recognizable to a being with a brain)... it'd take forever to even get past the first level. Plus the further in the game you want to expore, the number of sequences grows exponentially. For a game like Final Fantasy 3 US, there's trillions upon trillions upon trillions of input sequences -- only a small handful of which will get you through the game.

Fx3

05-17-2005, 10:57 PM

> There's more to emulating the system than just the CPU --
> and those other aspects of the system can greatly affect the
> CPU's execution, making straight code tracing difficult
> (player input, PPU/GPU status, system/mapper/other hardware
> IRQs -- even the sound system in extreme cases [*cough*
> Mig-29 Soviet Fighter])

Yes, Disch. You're right. :)
Go figure - you're still replying to this topic??? :|

>1. Open the rom
Yes, 'open the rom'... :) Actually, you LOAD the rom data.

>2. Locate code
>Once the ROM is opened, and you begin an Assembly Code dump, >the program locates it and will wait for the next instruction. Once >it is found, it will notify the user and provide a Hex address for the >code location.

I don't know, but it might be like the NES. Anyway, it seems you don't know a word of ASM 65816 for writting such statement "and will wait for the next instruction". :|

>3. Execute the ROM and search for ASM processes
Wow... so what's being done up to present moment???

>4. Log a list of Mnemonic calls used by the game
This is called "CPU emulator". The game can or cannot use ALL the instructions (or mnemonics)... Funny.

>5. Log processes in different text files by description
It's not that way; you need to identify subroutines and what they do in the game. Usually, the game has a main loop that calls the subroutines. You said about making drivers for specific games... go hack EVERY game, go analyse EVERY game.

MooglyGuy

05-17-2005, 10:59 PM

> In any case, it's a great idea. Definitely something I'm
> going to put into alot of thought.

...But going to get someone else to do all of the work upon.

king killa

05-17-2005, 11:14 PM

I like the term "plugins" a lot more in this situation. that's just me though.

<img src=http://neohjuggalos.com/fullhouse.jpg>

MooglyGuy

05-17-2005, 11:46 PM

In theory, you could just automatically step through the instructions and recursively follow jumps, which would eliminate the need for input. However, this doesn't work on systems that do not execute directly off the cartridge's memory space. That is, pretty much all systems, from the NES onward, given that the ROM on the cartridge (apart from Mapper 0) is banked into the main memory space and is not executed directly off the cartridge, so any jumps you'd encounter when stepping through the ROM would be invalid unless you were to figure out the banking structure of the ROM and assume each bank is at a given location - not an easy task. Furthermore, this is even more difficult for systems such as the N64, which can load code off the cartridge into an arbitrary place in main RAM (to whit, you can't execute anything in the N64's cartridge, you have to DMA it into main RAM first), or any disc-based systems which function essentially the same way.

Blade556

05-18-2005, 12:37 AM

> ...But going to get someone else to do all of the work upon.

For MONEY <img src=smilies/magbiggrin.gif>

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

hcs

05-18-2005, 12:45 AM

I was going to point out recursively taking branches, but you beat me to it...
The only issue I can think of would be when conditional branches are always taken, there's a chance that non-code or code that shouldn't be run at that point might be next. I'm told Super Mario Brothers has instances of this. I myself used it whenever I could with Escape from Pong (the absence of an uncoditional branch on the 6502 is irritating, who would want to waste a byte on an absolute jump?)

> (to whit, you can't execute anything in the N64's cartridge, you have to
> DMA it into main RAM first)

I've wondered if this is in fact true... the ROM is mapped to memory and can be read but this is always said to be "not recommended", if the CPU can load data from there why not code? Next time I'm in an N64 mood I'll give it a try.

Regardless of the format that the ROM is in and how the system needs to load it, if you're running the code in an emulator everything should be taken care of (assuming the emulator works with the particular game in question). This is the only way I can see this idea working, modify an accurate emulator to perform all the logging and tracing you want.

-http://www.halleyscometsoftware.com/hcs</a>

Blade556

05-18-2005, 12:47 AM

Meh....This is what I get for trying to find constructive criticism on a message board.

I see no reason why any of this should be impossible. I'm going to program this one way or the other. I'll learn C++ if I have to, I don't care at this point.

It will happen. To the doubters, <img src=smilies/moon.gif>. Any good thoughts are still welcome.

http://img.photobucket.com/albums/v227/DrShmoo/colorisin.jpg

Disch

05-18-2005, 01:08 AM

> In theory, you could just automatically step through the
> instructions and recursively follow jumps, which would
> eliminate the need for input.

I've seen code which takes advantage of branching ops to save a byte over JMP commands. For example I've seen a game BEQ back to some other area, but it's known to the game that Z will always be set (however it isn't set by the previous instruction). The data following the BEQ is actualliy game data and not executable. Such a situation is probably semi-common on the NES, and would pose a problem for taking both paths on a brach.

But of course indirect jumps pose the biggest problem -- and are the main setback for existing "smart disassemblers". This is why emus like aNESe and FCEUXD have taken the other route and say "ok, have the user actually play the whole game in the emulator, and mark which bytes are code and which are data, and this info will be used when you disassemble it later". This method relies on the player to expose the game code on their own, but provides an otherwise effective and error proof method of disassembling. Automating this process would be very difficult as previously mentioned.

Another style I can see being very problematic for such a disassembler (again another NES example since that's what I'm most familiar with), is a game writing a jump-to address to RAM, enabling NMIs, then doing an infinite loop to wait for VBlank. The NMI handler would then indirect jump to the previously written address. Such an address would be very difficult for a disassembler to track down effectively (or at all). However I'm not sure if any games actually do this method (though I would probalby use this method if I were interested in nesdev).

MooglyGuy

05-18-2005, 02:00 AM

> I've wondered if this is in fact true... the ROM is mapped
> to memory and can be read but this is always said to be "not
> recommended", if the CPU can load data from there why not
> code? Next time I'm in an N64 mood I'll give it a try.

Theoretically, you can, but IIRC the main reason why this is not done is because the contents of an N64's ROM are usually compressed, and they're usually decompressed upon loading into main RAM. Additionally, I believe that it's a bit slow to run directly out of cart ROM, but I could be wrong.

> Regardless of the format that the ROM is in and how the
> system needs to load it, if you're running the code in an
> emulator everything should be taken care of (assuming the
> This is the only way I can see this idea working, modify an
> accurate emulator to perform all the logging and tracing you
> want.

Sadly, there exists none for the N64. People can as much as they want, HLEing the RSP, RDP and audio hardware is not accurate. The only truly 100% accurate method would be to figure out the RDP's packet format (very easy given the patent documents that fully describe all possible RDP functions) and render everything via software rendering back into main RAM, and display that in a fixed 320x240 (or whatever resolution the game is supposed to use) window (or full-screen). Unfortunately, nobody wants to do this, even though expert opinion states that with a good R4300 recompiler, it shouldn't require more than 2GHz to achieve full-speed. On the other hand, even the most exact N64 emulator that one could possibly make would still use a small measure of HLE, as it'd be hard to extract the PI's internal ROM, which actually boots the machine and DMAs the initial cart code into RAM.

hcs

05-18-2005, 08:33 AM

> The only truly 100% accurate method would be to
> figure out the RDP's packet format

I've got the RDP manual right here if you want to give it a shot, with pixel-by-pixel descriptions and everything.

> as it'd be hard to extract the PI's
> internal ROM, which actually boots the machine and DMAs the
> initial cart code into RAM.

There's a clever solution to this, which someone working on that N64-based MAME driver pointed out:
1) clearly the PI boot ROM must be accessible to the CPU on boot or else it wouldn't be able to boot, but there's a switch of some sort that's thrown after the boot sequence is done that makes it unreadable (as this contains the CRC routine for the cartridge boot code and interfaces the CIC chip Nintendo doesn't want it read)
2) this switch must be reset upon a hardware reset so the thing can reboot
3) RAM is not cleared on reset, neither are the Watch regs
4) thus: write a routine to read out the contents of the PI boot ROM, stick it in RAM somewheres, set a watch exception on the first byte of the boot ROM and have it point at this
5) reset the consolue; it should hit the watch exception, read out the PI ROM, then return to let things finish booting. read the dumped ROM from RAM by regular means.

Now something that I don't think anyone emulates is the CIC itself, but I don't know enough about the emulation of protected games (like Banjo-Tooie) to know how this is currently handled vs. the ideal. Since there's only a finite number of CICs, though, games are happy for the most part when you just set the registers to a known state when beginning execution (the state which they'd be in after returning from the boot code).

-http://www.halleyscometsoftware.com/hcs</a>