123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183 |
- ################################
- # EQ2Emulator PacketParser.exe #
- ################################
- Contents
- --------
- 1. Legal and Disclaimers
- 2. What is PacketParser.exe?
- 3. Before you begin
- 4. Parsing your logs
- 5. Populating your EQ2Emulator database
- 6. Conclusion
- 1. Legal and Disclaimers
- =======================================
- There is nothing "Legal" or "Illegal" about using this tool as it is licensed under GPL, so
- don't hack it, reverse-engineer it, and re-release it calling it your own great tool. The
- only thing I want to really emphasize here is, if you use this tool to parse logs and populate
- your database, YOU ARE 100% RESPONSIBLE FOR THE OUTCOME.
- In no way, shape, or form is EQ2Emulator.net, developers, or database designers responsible
- for you losing data, or encountering an inability to parse and populate properly. This tool,
- like every other aspect of EQ2Emulator, is in a constant state of change and update, and at
- times will cease to function. However, it is not a high priority for the EQ2Emulator team to
- get it working again.
- In other words, be patient, report problems in a mature, friendly,
- respectful manner, and the problems will be resolved and a new parser package released.
- 2. What is PacketParser.exe?
- =======================================
- PacketParser.exe is the small utility that analyzes your collected packets from EQ2 Live game
- and turns them into "raw" data for use in building your world. This tool is only needed if you
- choose to build worlds based on the EQ2 Live experience. You can use this tool to parse Spawns,
- Items, Merchant lists, Factions, Spell and Collection data, and Quest/Conversation from the live
- game, with additional support added during the EQ2Emulator Alpha development cycle.
- In order to use PacketParser.exe, you of course need to have collected a log from the live game.
- You can collect the game data yourself using EQ2PacketCollector.exe - a separate tool (and thus
- documented separately). EQ2PacketCollector.exe is no longer available freely to the public, but
- you may PM John Adams or LethalEncounter on the EQ2Emulator.net forums to receive a copy of the
- program.
- 3. Before You Begin
- =======================================
- Before you can parse a packet collect, you should have a local instance of MySQL 5.x installed.
- You can have MySQL on your Windows XP workstation, or somewhere on your network - it does not have
- to be on the same machine... but your database should not be located via a WAN or over the internet.
- It would be far too slow!
- Create a folder, example C:\eq2parser, and put the following files into that folder:
- Required:
- - PacketParser.exe
- - zlib.dll (downloadable package - search for it on the internet)
- - WorldStructs.xml
- - SpawnStructs.xml
- - ItemStructs.xml
- - parser_db.ini
- - Parser DB.sql (not needed except to source in raw DB structures)
- Optional (if you wanted to script some mysql actions)
- - parse-all.bat
- - mysql.exe
- - mysqldump.exe
- MySQL Database setup:
- Assuming you use SQLYog or some other MySQL GUI Tools (google them, they are your friend), you must
- connect to your MySQL instance and create a blank database. Refer to MySQL documentation on creating
- a database. Example database name for the parser: eq2_rawdata.
- "Source" in the included Parser_DB.sql file - by this we mean open this Query and execute it so it
- will build the database structure required by PacketParser.exe - without this, you are going nowhere.
- If you encounter errors, you need to explore how to get around them as this is not a lesson in becoming
- a Database Administrator. :)
- Once your database is created and you see tables like "raw_spawns", "raw_items", "raw_spells" etc, you
- are almost ready to go.
- NOTE: Since PacketParser.exe is now a "controlled release", updates to both the EXE and Parser DB.sql
- will usually be posted in In-House Tools, under PacketParser.
- parser_db.ini:
- Edit parser_db.ini to reflect your database settings. Here is an example:
- [Database]
- host=localhost
- user=root
- password=sekr3t
- database=eq2_rawdata
- host: the IP or FQDN (fully qualified domain name) of your MySQL server. If it is on the same machine,
- use "localhost"
- user/password: This is a MySQL account that has SELECT, INSERT, UPDATE, and DELETE rights to your
- eq2_rawdata database.
- database: The name of your raw database - in my example, eq2_rawdata.
- 4. Parsing Your Logs
- =======================================
- Included in your PacketParser package is a sample BATCH file that is designed to identify *.log files
- in the current directory and execute the PacketParser.exe (with parameters) on each file individually.
- Once the parser is done with the file, it will move it off to a "Done" folder. This is to prevent the
- same log from being re-parsed should you have to stop and restart the job.
- Move your log(s) into your C:\eq2parser directory with your other parser files, and simply run the
- included "parse-all.bat" file. This batch files is executing the following command:
- - PacketParser.exe -spawns -items -spells -quests -factions -collections {log_file}
- You can run the command-line version of the PacketParser.exe yourself - you do not need to use the
- batch file. Run PacketParser.exe and hit ENTER to see a list of command-line options.
- Example, if you just want to parse Spell information from your logs, run:
- - PacketParser.exe -spells {log_file}
- * Replace {log_file} with the name of your collected log (ie., EQ2PacketLog.log)
- You can stack the command-line options as many or as few as you like. There has to be at least one,
- and for PARSING logs, -populate is not valid (see next chapter).
- NOTE: The parse-all.bat file is setup to output parser data to a parser_log.txt file. This is the
- troubleshooting information we would need to see if you are having problems. Simply post your problem
- and attach that text file. If it is larger than 10kb, you probably should ZIP/RAR it first.
- 5. Populating your EQ2Emulator Database
- =======================================
- During the 0.7.0 development cycle of EQ2Emulator, some critical changes have been made to the database
- structures regarding Items, Spawns, Spells and Quests. Take note of these as you continue populating
- your worlds using our tools. Recently, major changes to the Spawns structure broke the -populate
- functionality, but a new parser was released to resolve this issue.
- To populate your EQ2Emulator World database, simply run the following command:
- PacketParser.exe -populate {eq2world_db}
- * Replace {eq2world_db} with the name of your database that EQ2World.exe uses
- The -populate command will move all raw Spawn data from parser DB to World DB for use in your world.
- There are a few other commands currently available:
- -loaditems - this will populate the Items tables in your World DB from Raw data
- -loadfactions - this will populate the Factions tables in your World DB from Raw data
- 6. Conclusion
- =======================================
- There is literally tons of RAW data coming from packet logs, some we do not even utilize yet in the -populate
- commands of Parser. Examples of some of this data is:
- raw_dialogs : These are the conversations you have with NPCs in game, with the clickable option buttons
- raw_conversations : These are the casual shouts NPCs give as you pass by or Hail them - no interaction
- raw_map_data : You should update your map_data table in your World DB with newly parsed raw_map_data as needed
- raw_merchant_items : As you parse Items, if they came from a merchant window, this table shows you what spawn_id had what item_id
- raw_quests : This data is what we use to build LUA scripts for quests - currently no automation available
- raw_sounds : These are your door squeaks and other sound effects in game, mapped to their raw_spawn ID
- raw_spells : This data is used to build our Spells lists, or update existing information with new captured data
- raw_zone_points : This is an attempt for the collector/parser to locate any known zone-in-points when you zone while collecting (not always accurate)
- There is so much more we can gleen from our logs we hvae yet to discover. If you feel like helping identify
- some of this data, feel free to join in on the conversations in Development and offer your help.
- Happy World Building!
- -John Adams
|