WorldSpawn official WIP thread

[MadGypsy]

Wow! You have a lot of questions. That's OK. It is just going to take me some time to answer them all. Let's start with some info.

I downloaded halflife and started ripping apart more wads. The code that I posted yesterday is only going to help you for entries.type = 0x43. Parsing 0x40,0x42 & 0x46 types is not properly handled in my code. This is because there are only 2 wad3 specs that I can find and they do not entirely agree. One of them illustrates a much more robust texture struct but it seems to be inaccurate. Following it to the T is not producing the proper results. If you know of some official wad3 specs somewhere there is absolutely no doubt that I could write a full featured wad3 parser. I have had no luck finding anything official. All that being said let me start answering your questions.

First of all I see you are using the old code from my github for flash quake. There is nothing wrong with that necessarily but, it is not optimal. If you want to "steal" that code you are more than welcome to or I can give you even better code to "steal". It's not stealing if I tell you flat out you can do whatever you want with it. Actually, I would love to see you do something with it. You don't even have to give me credit cause, credit + a nickel still won't buy me a piece of gum.

If you've been using my code then you know my content manager consists of 4 parts. Importer, EventHandler, Library and ContentManager. The Importer triggers the EventManager which loads the file and bubbles the resulting event back up to the Importer. The Importer is then triggered (if Event.COMPLETE) to switch/case the file extension and format the loaded bytes as either ByteArray, BitmapData or String. Once formatted it is stored in the Library. Each filetype has a getter in Content(Manager) which allows you to retrieve the desired data by name. The main reason for the Importer and Content(Manager) is to create a public interface that abstracts the EventManager and Library allowing both to remain encapsulated as internal classes.

I guess one way to answer your ContentManager questions is simply to say I just changed the names and namespace of the stuff you are already using. I did this because I also stripped the files down to bare minimums to deal with wad because, I have every intention of giving you all of the current code, and I wanted to make it as simple as possible to read/use.

@Electron or NW.js, BoxedApp Packer or Engima Protector

I don't know what any of that is. Honestly, all of that seems way too cumbersome and overblown just to make flash stuff and/or package it as a standalone. I use flashDevelop for EVERYTHING.

@i don't know what to do with haxe.zip

here is my old cheater script. The only things that you do not have is utils.ByteStream because I invented it and Importer cause I invented that too. Simply replace those with haxe.io.BytesInput class and some other class for managing storage of the bytes.

package parsers.zip;

import utils.bytes.ByteStream;

import haxe.io.Bytes;

class Zip
{
	private var _entries:List<haxe.zip.Entry>;
	
	private var _imported:Map<String, Bool>;
	
	//don't think I even need this. I certainly don't need it "yet"
	@:isVar public var entries(get, never):List<haxe.zip.Entry>;
	public function get_entries():List<haxe.zip.Entry> return _entries;
	
	/**	CLASS_CONSTRUCTOR
	 * @param	data - zip data as ByteStream from Import
	 */
	public function new(data:ByteStream):Void
	{	
		var reader = new haxe.zip.Reader( data.input );
		_entries = reader.read();
		
		setImported();
	}
	
	/** SET_IMPORTED
	 * 		init false for all entries. 
	 * 		true indicates that the entry has already been decompressed to the local lib 
	 */
	private function setImported():Void
	{	if (_imported == null) _imported = new Map<String, Bool>();
		
		var extension:String;
		var filename:Array<String>;
		var name:String;
		
		for (entry in _entries)
		{
			filename	= ( entry.fileName.split("/").pop() ).split(".");
			extension	= ( filename.pop() ).toLowerCase();	
			name		= ( filename.pop() ).toLowerCase();
			
			_imported.set(name+extension, false);
		}
	}
	
	/** LOOP_RETRIEVE_AN_ENTITY
	 * @param	byName	 - file name
	 * @param	byRegExt - regex of possible extensions for this file - ie ~/png|jpg|gif|jpeg/i
	 * @return	zip entry struct
	 */
	public function getEntry(byName:String, byRegExt:EReg):haxe.zip.Entry
	{
		var filename:EReg = new EReg(byName, "i");
		for (entry in _entries)
			if (filename.match(entry.fileName))
				if(byRegExt.match(entry.fileName))
					return entry;
		
		return null;
	}
	
	/** MAKE_ZIPFILE_LOCAL
	 * @param	byName	 - file name
	 * @param	byRegExt - regex of possible extensions for this file - ie ~/png|tga|gif|bmp/i
	 */
	public function setFile(byName:String, byRegExt:EReg):Bool
	{
		var entry = getEntry(byName, byRegExt);
		byName = StringTools.replace(byName, "#", "*");
		
		if(byRegExt.match(entry.fileName))
		{	var byExt = byRegExt.matched(0);
		
			if(!_imported.get(byName+byExt))
			{	_imported.set(byName+byExt, true);
				
				if (entry != null) {
					Importer.libBytes(byName, byExt, new ByteStream(cast uncompress(entry.data).getData()));
					return true;
				}
			}
		}
		
		return false;
	}
		
	/**	UNCOMPRESS_BYTES
	 * @param	bytes	- compressed Bytes
	 * @return	uncompressed bytes
	 */
	private function uncompress( bytes:Bytes ):Bytes
	{
		var tmp = haxe.io.Bytes.alloc(65536);
		var out = new haxe.io.BytesBuffer();
		var z = new haxe.zip.InflateImpl( new haxe.io.BytesInput(bytes), false, false );
		while( true ) {
			var n = z.readBytes( tmp, 0, 65536 );
			out.addBytes( tmp, 0, n );
			if( n < 65536 )
				break;
		}
		
		return out.getBytes();
	}
}

I will tell you, if you intend to develop in Haxe for windows (cpp) you are going to need to drop back a few revisions from the most current. There is a bug that makes msvc2010 hang on the resources part of the compile and when it does finally build it's broken. I guarantee you that shit will be fixed soon cause I have found a LOT of threads with people complaining about it directly to the people in control of fixing this. I fixed it myself but it is hacky and temporary til they fix it so I can at least keep working.

I'm sure I have left some questions unanswered. I need a break. I'll be back in a bit, look at your post again and try to answer anything I missed. You are welcome to ask more questions before I finish answering the ones you already asked. I'll just keep chipping away at them til I catch up.

EDIT: One problem I can see you having regarding "then" vs "now" is that I am constantly upgrading my skills as a programmer. Code you are reading from last year is going to be "junky" in comparison to my code now. Code from right now will be "junky" in comparison to my code of the future. My real life goal has always been to be an amazing programmer. I realize I probably already am that but it is not good enough for me. If I already am amazing then the new goal is to be phenomenal. When I get to phenomenal the goal will be to be ... something better than phenomenal...lol. My point being that you are asking me about code that I have moved well beyond at this point and for me to answer you takes away from the much better answers I could give you regarding the present.

Allow me to give you an example. In the past I would program by the seat of my pants with absolutely no direction beyond some vague notion of a goal. These days I spend maybe a few hours a week writing spreadsheets in OpenOffice that map code I intend to write. The objects in my spreadsheets reference other spreadsheets that go into even deeper detail. I used to spend crazy amounts of time toiling over code and mistakes. It is 100% factual that I am practically famous for programming 20+ hours straight. These days, I follow my spreadsheets for 2 or 3 hours and get substantially more work done with far far far less errors.

[MadGypsy]

@format.zip or other?

lol, if you look at format.zip classes it says something like

#if haxe > 3.2
haxe.zip.Zip
#else
...custom code...
#end

in other words, using format.zip is just redirecting you to haxe.zip. As a matter of fact you can pretty much dump format altogether. It's unfinished and pretty much obsoleted at this point. openfl.display.Loader can handle jpg, gif(first frame) & png. openfl.net.URLLoader can handle text, bytes and urlvariables. There are zip and lzma packages built into haxe. Between all of that you should be able to open just about anything. I wrote my own classes for these things and more because parsing is my "thing" but, I can't say with any authority that my methods are any better than the ones built into haxe and openfl. As a matter of fact, there is no reason to believe that it's even possible for my way to be better. I'm using haxe packages to create my parsers and so is openfl and in one case I'm using haxe packages to write a custom parser that already exists in the haxe core.

---

I see your code structure seems very determined to work with zips at some point. In my experience, there is pretty much no gain from this. All you are really doing is adding a bunch more processes to the load. I understand it allows you to keep your file-structure compact but, how important is that really. Think of this... do you think users appreciate longer load times for every level because they have a compact source or do you think they really don't give a shit how tidy it is if you can make it fast? The shortest route between 2 points is a straight line. Using a zip is like bending that line because everything you need from it now needs to be uncompressed before you can work with it. Also, using a zip means that you will more than likely be loading a bunch of stuff that won't even be used. I mean, unless you intend to independently zip the resources for each level there is definitely going to be a bunch of wasted loading. In contrast, independently zipping the resources would mean that you would have a bunch of duplicate resources... unless everything will be completely different for each level.

[MadGypsy]

I found this c# source for parsing wad3. It appears to be proper. Imma port it over to AS3 and then port it again to HaXe or maybe I will do it the other way around. Either way, if that source isn't some bullshit I'll have a wad3 parser licked in an hour or so. Imma go do all of that right now.

I will also rewire it for you so that instead of you putting in a name and the WAD3 class retrieving the data... and then saving the data to the library, it will instead require the bytes as an argument and return a vector of wad3 results. This way the script will be standalone for you.

side note: I know that I have made it pretty clear that I am basically a haxe-only programmer at this point but, I want to thank you for inspiring me to mess around with some AS3. The total ease that I program in AS3 with has made me realize I am still not completely "at home" with HaXe. I'm not doing bad with it but, my HaXe programming is with noticeably more effort.

[SourceSkyBoxer]

I guess one way to answer your ContentManager questions is simply to say I just changed the names and namespace of the stuff you are already using. I did this because I also stripped the files down to bare minimums to deal with wad because, I have every intention of giving you all of the current code, and I wanted to make it as simple as possible to read/use.

No I didn't get your github server. I use "JPEX FREE FLASH DECOMPILER
You can download and open SWF file and it generates decompiled SWF into AS3 scripts. But it is not best decompiler - just it looks like "Spying to AS3"
Free Decompiler

@Electron or NW.js, BoxedApp Packer or Engima Protector

Electron and NW.js are universal wrapper like Chrome Apps ( HTML5, JS, Full WebGL support with PPAPI FLash Player support. )
Electron
NW.js

BoxedApp Packer and Enigma Projector are like exe-packer for example
Both are fast packing and generating into single executable example:
normal version:

assets/openfl.png
manifest/default.json
icon.ico
MyGame.exe
lime.ndll
regexp.dll
std.dll
zlib.dll

Embedded / bundled version:

assets/openfl.png
MyGameEmbed.exe or MyGameBundled.exe

BoxedApp Packer
Enigma Protector
If you want get fullversion Enigma VirtualBox but you cannot use commandline like

MyGameEmbed.exe -game base -width 1024 -height 768

Remember BoxedApp Packer or Enigma Protector to have with commandline. If you want download test version of BoxedApp Packer and you can try commandline.
Enigma Virtual Box

I will tell you, if you intend to develop in Haxe for windows (cpp) you are going to need to drop back a few revisions from the most current. There is a bug that makes msvc2010 hang on the resources part of the compile and when it does finally build it's broken. I guarantee you that shit will be fixed soon cause I have found a LOT of threads with people complaining about it directly to the people in control of fixing this. I fixed it myself but it is hacky and temporary til they fix it so I can at least keep working.

Lol I am using Visual Studio 2015 / 2017 Community - no problem with Haxe HXCPP. But It is sad because Lime from Github I already tried to compile into static version like example "openfl build cpp -static". I don't need to get dlls and ndlls of compiled executable like this:
normal version:

assets/openfl.png
manifest/default.json
icon.ico
MyGame.exe
lime.ndll
regexp.dll
std.dll
zlib.dll

static version:

assets/openfl.png
manifest/default.json
MyGame.exe ( bundled / embedded with regexp.lib, std.lib, zlib.lib and lime.lib )

If you download lime from Github and compile with example "lime rebuild windows -static" than it has always warnings with 2 libs of regexp and std. than you type openfl build cpp windows -static and gets 2 errors. We need generate normal version than we would like to use alternative with BoxedApp Packer or Enigma Protector or Enigma Virtual Box.

here is my old cheater script. The only things that you do not have is utils.ByteStream because I invented it and Importer cause I invented that too. Simply replace those with haxe.io.BytesInput class and some other class for managing storage of the bytes.

...

@format.zip or other?

lol, if you look at format.zip classes it says something like

#if haxe > 3.2
haxe.zip.Zip
#else
...custom code...
#end

Yeah But it looks complicated. I will try...

If we create simple like deng.fzip found from GithubPS. You already downloaded and edited with "exists(path:String):Boolean" lol
I think if I will try to create easy zip-alternative for haxe.

I see your code structure seems very determined to work with zips at some point. In my experience, there is pretty much no gain from this. All you are really doing is adding a bunch more processes to the load. I understand it allows you to keep your file-structure compact but, how important is that really. Think of this... do you think users appreciate longer load times for every level because they have a compact source or do you think they really don't give a shit how tidy it is if you can make it fast? The shortest route between 2 points is a straight line. Using a zip is like bending that line because everything you need from it now needs to be uncompressed before you can work with it. Also, using a zip means that you will more than likely be loading a bunch of stuff that won't even be used. I mean, unless you intend to independently zip the resources for each level there is definitely going to be a bunch of wasted loading. In contrast, independently zipping the resources would mean that you would have a bunch of duplicate resources... unless everything will be completely different for each level.

But I don't know what do I Content.hx / ContentManager.hx because I tried example code of your code but I found missing haxe code. like this.
Example:

var test:Bool = Wad.parse(Content.wadfile("halflife"));

it is haxe code.
I already write Wad.hx and I don't write "Content.hx" - I create new haxe code and I try to formulate. If I don't know structure of Content.hx.

Thanks for understanding! I hope I have experience or idea of structure than I can write....

[SourceSkyBoxer]

PS: For rebuilding with lime static - Now it has not errors. yay


and HelloWorld with static

How do I get?
Write command lines:

git clone --recursive https://github.com/openfl/lime

Wait until download process is completed.
Make lime from github as development version

haxelib dev lime lime

Than you can rebuild it. Make sure format from haxelib database while it is installed. And please download Visual Studio 2017 Community because it works for static fine. Old version cannot because Visual Studio 2010 is very incompatibly. Since it happens errors if I use openfl build cpp -static and lime build cpp -static with Visual Studio 2010 for Windows 7, 8 and 10. But old version of Windows 7 without Service Pack. or SP 1 - I don't remember - I got success.... with Lime 2.9.0 and OpenFL 3.6.0 with Visual Studio 2010 without errors. Now it is very incompatibly. That is why we need use last version.
Rebuild hxcpp if lib files don't exist than you need rebuild

lime rebuild windows hxcpp

lime rebuild windows -static -64 -release -clean

Wait until ndll and lib!
Don't worry! warnings are okay now.
Complete with tools:

lime rebuild tools

Now you can try and path to your WorldspawnBSP

cd WorldspawnBSP

openfl build cpp -static -clean

( If you want clean up generated builds ) -> -clean

Than you're happy with standalone bundled WorldspawnBSP.exe without dlls and ndll
If you don't want embed bsp and textures, models etc into executable.
just open and edit with project.xml
<assets path="/"/> just root directory of generated application.
example

WorldspawnBSP.exe
game/
  textures/
  maps/e1m1.bsp
  ...

I hope you have not problem with static version of haxe app.
We don't need to use BoxedApp Packer, Enigma Protector or Enigma Virtual Box.

[MadGypsy]

Awesome job!!!

Really! You just did an awesome job! Where did you come from? You start off seeming a little green with a wad parser and then you post everything about making HaXe your slave via the command line. I'm blown away.

[SourceSkyBoxer]

Awesome job!!!

Really! You just did an awesome job! Where did you come from? You start off seeming a little green with a wad parser and then you post everything about making HaXe your slave via the command line. I'm blown away.

Thanks my best I come from Germany.

Yeah I am not finish to create wadparse. I will write...

[MadGypsy]

@wad parse

I just need maybe 30 minutes behind my computer to finish my port of that script I linked last night. It's 11 here I'll be home 7:30 ish .... Add in a little decompress time and I would say I should have it by 9.

Let me do this for you. I really will do it...top priority. The ONLY hitch is that maybe that c# script doesn't work. If so then my port won't either. I went over the script last night, understand it in full, and am pretty confident it does work, though.

[MadGypsy]

Are you proficient in HaXe programming? I don't mean just writing code that is syntactically proper. I mean in a much deeper sense, like creating abstracts, working with the ADT enums and macros, compiler conditions related to target and api versions... etc..

If so, something tells me that you may be the only person I have ever corresponded with in my life that would actually be useful to team up with for certain things.

By certain things I mean specific things. I'm not really looking for a partner regarding my engine BUT I could see where releasing certain aspects of it to you would be beneficial. In contrast, your engine will require things that I've already done. I believe half-life uses bsp30. I've already parsed bsp29, bsp2 & 2psb. There is no reason to believe bsp30 would be any great challenge. I also have written numerous mini API's that make HaXe programming easier. To share some of this with you would be no big deal.

I'm just throwing this out there. We have skills and knowledge that is hard to find other people with, and often when you do find those people they aren't very good. I've been programming like 20 years or something. I didn't come into HaXe green. I was programming well beyond "hello world" in HaXe from minute one and I have spent about a year working my ass off to straight up master it. I guess my point is (by observation) you and I are about as close as either of us are going to get to an actual peer, if your answer to my first question is at least leaning towards a yes.

I believe we could be very useful to one another.

[SourceSkyBoxer]

Hello Michael,

Keep cool! I am trying to library - If I will check like custom zip like nochum.zip or deng.fzip


I resolve solution of unsupported describeTypes into custom function like this

/**
 * unsupported describeTypes
 */
private static function isUncompress(data:ByteArray):Bool
{
	data.uncompress(CompressionAlgorithm.DEFLATE);
	if (data.length > 0)
	{
		return data.length > 0;
	}
}

private static function isInflate(data:ByteArray):Bool
{
	data.inflate();
	if (data.length > 0)
	{
		return data.length > 0;
	}
}

It is for deng.fzip.ZipFile.as into net.sourceskyboxer.ZZipFile.hx. You know problem of as3hx can not convert that is why I need redefine. I try to convert to haxe with reading and writing zip without haxe.zip ( Do you think it can not work or it can work? If not than I use with haxe.zip ) If I get success or problem.

// EDIT: Is it correct or wrong haxe.crypto.Crc32.make() is like ChecksumUtil.Crc32() method?
Or I should use your post of example code zip. I am sorry - I understand bit hard with haxe.zip I will try until whole Zip, ZipFIle, ZipLibrary and ZipEvent..

Thanks sorry for late because I got bad weather in Germany. My daddy says me. I need switch off my computer. Because it is safe. Thanks for understanding!

[MadGypsy]

I mean ABSOLUTELY no disrespect but, it makes me giggle that your code is in broken English...

HAS_INFLATE

@Do you think it can not work or it can work?

Of course it can work but, think... you are just going to end up rewriting (to some degree) what is already provided by HaXe. Which is more or less what I did. It's not bad but, you need to weigh how important it is to you.

[MadGypsy]

Heads up: got home 30 minutes early and I'm not going to decompress or relax at all. Straight to wad3 parsing. Be done soon. Stay tuned.

edit: ported all of it and there are issues. Probably because this dudes code is chicken scratch spaghetti. I need to strip it down to the stuff that matters. However, Doctor Who is on and I watch this stupid shit so, I'll resume when it's over.

[SourceSkyBoxer]

Heads up: got home 30 minutes early and I'm not going to decompress or relax at all. Straight to wad3 parsing. Be done soon. Stay tuned.

edit: ported all of it and there are issues. Probably because this dudes code is chicken scratch spaghetti. I need to strip it down to the stuff that matters. However, Doctor Who is on and I watch this stupid shit so, I'll resume when it's over.

Yeah it is hard to read because wad is not zip and it is special "texture package" like Quake has pk3 or pak.

You're right.
I have tried no success.
Is it correct like your Content.as for getBitmapData static method:

public static function getBitmapData(byName:String):BitmapData
{
	_loader = new Loader();
	_bitmapData = new BitmapData(0, 0);
	_loader.contentLoaderInfo.addEventListener(Event.COMPLETE, onCompletedHandler);
	if(ZIPLIB)
	{
		if(_zip.exists(byName))
		{
			if(ZIPLIB)
			{
				_loader.loadBytes(_zip.getFileByName(byName).content);
			}
		}
		_loader.load(new URLRequest(byName));
	}
	
	return _bitmapData;
}

private static function onCompletedHandler(ev:Event):void
{
	_bitmapData.draw(ev.target.content);
}

Is it correct or wrong?

I found google But I am not sure. If it is Event.Init or Event.Complete for drawing with loader ( ev.target.content )

And Wad.parse of your code. But I am not sure. If I will try next day. Because it is hard to read and to load to Bitmap. I know Quake has not problem. But Half-Life makes hard. I will find Wad lumps and bsp lumps from Valve Developer website. But contents of Half-Life are removed since 2002.
Half-Life BSP
Wad3
It looks very close to MIPTEX of Quake.
Half-Life Sprite Sorry only German Language. Just check only code with English version.

Test for wad3 and bitmap:

package
{
	import com.valve.halflife.ContentManager;
	import com.valve.halflife.Wad;
	
	import flash.display.Bitmap;
	import flash.display.Sprite;
	import flash.display.StageAlign;
	import flash.display.StageScaleMode;
	
	public class WadView extends Sprite
	{		
		public function WadView()
		{
			stage.align = StageAlign.TOP_LEFT;
			stage.scaleMode = StageScaleMode.NO_SCALE;
			Wad.parse("halflife", MyFunc);
		}
		
		private function MyFunc():void
		{
			var bm:Bitmap = new Bitmap(ContentManager.getBitmapData("AAATRIGGER"));
			bm.x = bm.y = 10;
			bm.scaleX = 4;
			bm.scaleY = 4;
			addChild(bm);
		}
	}
}

Nothing shows. I already check halflife.wad into current bin directory.

I think I will resolve next time.

[MadGypsy]

OK, I'm pretty sure that I finally have it. There is just a little more that I want to do because my return data is too anonymous, I skipped all mips and I skipped fonts too. I just need to reformat how I am saving the results a bit and "unskip" mips and fonts. I am pretty sure that what I have right now is exactly correct though.

@SSBoxer

I will answer your questions in a bit. I will tell you though that bitmap.draw is a TERRIBLE way to do that.I will also tell you that your code is not reusable. What are you going to do... code a loader for every file type? I will give you my content manager in a stripped down way and you will see amuch better way to do it.

//so you understand how the byte array should be formatted
var someByteArray:ByteArray = new ByteArray();
someByteArray.endian = Endian.LITTLE_ENDIAN;
someByteArray.writeUnsignedInt(0xff000000);
someByteArray.writeUnsignedInt(0xffffffff);

//how to draw it
var bmd:BitmapData = new BitmapData(2, 1, true); //(width, height, allow transparent pixels)
someByteArray.position = 0; // if you don't do this EOF error
bmd.setPixels(bmd.rect, someByteArray);

results: a black pixel and a white pixel

ALSO - the loader already contains the bitmap data... Loader is a display object

Bitmap(ev.target.content).bitmapData //cast content to Bitmap and use Bitmap.bitmapData to retrieve

Whatever you do COMPLETELY FORGET ABOUT .draw() FOREVER. It is the worst way to draw images. If you had (ex) 200 images to draw it will take the rest of your life to complete.

[MadGypsy]

So you can see I am dead serious. I scaled it up a lot because it was an itty-bitty image and I wanted you to see every pixel. I told you I was gonna do it and I meant that! I need a little more time but, this is a done deal. This was fun. I ported a script from a language I don't know at all...pbbbt. If that doesn't show the naysayers what I have been saying all along about programming in ANY (humanly readable scripting) language, I'll find something more impressive later.