WorldSpawn official WIP thread

[MadGypsy]

Macros, macros, macros...

I finished up a bunch of stuff and then decided to go find all the reasons why it's "bad". I already knew what I was doing was not optimal but, I didn't know why... didn't really care. It took some searching cause this shit is so poorly documented but, I finally found some Google group talking about it. In short, using reflection is bad because of dynamic vs static targets. In order to make one code harmonious for both target types involves doing a bunch of bloated background stuff and, as the use of reflection becomes more frequent it can actually cause performance issues (hooray to me for making it as easy as possible to have terrible performance :pats self on back: )

The solution is macros. What is a macro, you may ask... Well, mostly, I have little to no real idea. I could show you one but, I haven't had a chance to really pick it apart yet. Some in the HaXe community refer to it as "dark magic" so, it's safe to assume I will be spinning my wheels and playing with lots of hornets while I master it. The good news is, this is the final frontier for me. I went over the HaXe docs (skim style) last night and macros are the only thing left that I couldn't write an essay on.

This means I am just a short time away from taking this education I have given myself and using it to properly make useful things. As always, I have no regrets regarding any time I have wasted working on things that aren't optimal. I didn't teach myself 20 some-odd languages by worrying about best practices and optimizations right out of the starting gates. I learned these languages by writing tons of terrible code and then realizing why it is terrible...

Need to make an order... Be back in a bit to finish my thoughts.

[MadGypsy]

Back...

...terrible code. I have always been of the mind that it is more important to lay a foundation of working code and then make it better by doing research, when first learning a language. I've also always been of the mind that making a virtual nothing is better than attempting a something, and the reason is simple... virtual nothing's force you to worry more about how you are programming and less about what you are programming. For instance, do I really give a shit if dynamics are keyable... not at all. I could have written a simple class that remaps dynamic to map and called it a day. What all this focus really did was put me in a position to "solve problems" and dig really deep on the various things that are available to me to accomplish the solution. In other words, it was a "discovery" exercise but, by treating it with all seriousness I kept forcing my brain to invent ways to use other possibilities to accomplish my goals... and much more was discovered. I believe it was Ben Franklin that said something to the effect of "On the contrary, I did not fail 100 times. I discovered 100 ways that do not work." That guy was the greatest inventor of his time and for generations to come.

If I stopped right now I would still be a good HaXe programmer. I intend to be a great one. I intend to get my avatar on the HaXe contributors page and go on to make sizable and notable contributions. I will master this language AND the underlying nuances of all the most common languages it ports to. One way I intend to do this is by eventually writing the HaXe package for QC. Nobody else is ever going to do it and I know plenty enough about QC to make it possible. I may be the only person in the world that knows enough about both (QC and haxe) to get it done at all. I do need to learn a bit more about HaXe hidden internals before taking this idea serious though. My plan is to rip apart haxe PHP internals cause I am the strongest in that language vs the other haxe transpile languages. If I can make all the connections with PHP I can recreate them with QC.

For the record I don't care if no one ever uses my haxe to QC package. It's not as much about creating something for interested developers as it is about taking a simple language that I mostly know and using it to connect all the final dots in haxe.

[MadGypsy]

I'm bored at work (super slow) and I was rereading some of my posts here. A recent one stated that I was too embarrassed to explain an oversight. Well, I'm still embarrassed but, I don't have anything else to do so...

My system relied on using enum constructors to strong type values. The one big oversight is that meant every single value is an enum. Consider...

Name:value

That would have become

Name:someEnum(value)

Now, if name was a key of MY object that wasn't a big deal cause I could reroute the @:arrayAccess to point to the enum constructor BUT what if name was a key of a "natural" type (ie map, array, etc) .... everything falls apart.

In short, I turned everything into enums and now that's what they are, systematically breaking everything except the keyable object I was inventing... genius! They should hire me at microsoft. I'd fit right in with terrible ideas like this.

[MadGypsy]

Holy shitcakes, Batman! Where the hell has this been in my thousands of searches for haxe tutorials/info/etc?

For those that don't care to click that: It's nothing but a fairly sizable cookbook for the haxe language. It's nothing but (probably) every example I ever needed. Psssh, I really wish I would have found that 11 months ago when it first went live. I've been using haxe APIs as a cookbook (of sorts) this entire time. There's nothing like trying to learn a language by "figuring it out" through observation. Oh shammalammadingdong... meh, I'm probably more bad-ass overall by doing it the hardest fucking way possible this whole time. Make no mistake though, you can call me Chef with all the recipes I'm about to mix.

::cracks knuckles::
::brews coffee::

It's about to be one of those don't sleep for 2 days situations that I'm famous for. I can sleep all I want when I die from this dumb shit I do to myself.

[MadGypsy]

Not literally cause I seriously need to learn this but,

FUCK MACROS!

I knew this was going to be really hard but this is so damn hard I almost feel like I might as well throw away everything I know about programming, wipe my mind clean and start completely over. Dark Magic indeed. It's not really all that funny that the haxe docs says something like "It's not magic and it's definitely not dark" but this other guy that is like a macro god or some shit told some beginner "good fucking luck" in response to him learning macros....

I think it may be time to befriend back2dos. He's basically the "spike" of this shit and I could really use a spike right about now. I'm faced with 2 options. Learn macros, fix my code accordingly and receive a bunch of optimizations or don't do that and live with somewhat bloated/slow code. I seriously can't see myself doing the latter but I'm 14 hours into this shit and I feel like I made about 2 steps of progress.

::cracks knuckles:: Fuck that, this shit is NOT going to defeat me. I'll never sleep again if that's what it takes to nail this!

edit: You know what the problem is?... Old dog/new tricks. ALL of my programming experience has been very similar. Objects are objects...arrays are arrays...etc and you just use them. You don't worry about highly complicated underground ways of doing things because the stuff is as good as it's gonna get. Haxe though, does not work like that. Sure you CAN "just use it" but there is a penalty and that penalty is directly related to how good you really want to be. In a sense Haxe allows you to be an "it works" programmer so it wont completely scare you away from even attempting to learn it but, then you get better and learn more and haxe basically says "forget all that 'it works' crap. Learn this far superior and incredibly powerful way that is nothing like anything you are used to". Macros, in short, are a way to write code that writes code. The macros I write are basically a preprocessor. I wish it was easy to do as it is to explain. Imagine if someone was teaching you how to walk. "You just put one leg in front of the other" but only problem is... you don't have any feet and you have no idea how to get some.

It may even be true to say that most programming languages try to remove you as much as possible from the lowest levels. I mean this in a writing script sense. Haxe however, goes the other way. Sure there is plenty of separation for most things regarding built-ins and what-not that make your code lighter but, not for objects, not if you want them to be optimal. It's practically a rule "never create an object and stay as far as you can from reflection". Macros seem to let you create object behavior without ever actually instantiating an object. Super tricky.

I'm almost afraid to go dig in the away3d source and realize that there are about 50 bazillion optimizations that could be made regarding macros. I honestly don't recall a bunch of object creation in the source but I also don't recall seeing a bunch of macros. The truth is, I know if I learn macros and realize that away3d is not using them I'm going to end up fixing as much of that api as possible. It's gonna take the rest of my life.

[MadGypsy]

Hah! Check out this stream of thoughts, observations and realizations that just happened in my head.

I took a mini break from macros for a couple of hours and did some investigation on objects. I came across the same info I already knew, objects are slower on static targets because a reverse look up needs to be performed every time you access a field. This time however, I had a different thought "What are all the static targets?". I came across this:



On a side note, I noticed flash and cpp are static but, javascript and neko are not. This made me wonder if that's why flash and cpp substantially outperform these targets but, that isn't necessary info to where I am going with this.

I also noticed that they listed the targets which are compatible with the filesystem package. That got me thinking about my content manager. My content manager uses openfl to pretty much "blanket solution" the filesystem issues for all targets. I really don't like that. In the past I would have boasted that I am writing code that doesn't have compiler conditions but, I'm really starting to lean the other way on that idea. I am more of the mind that I would like to eliminate my sparse use of openfl and then eliminate openfl. I'm using this massive library just to avoid some compiler conditions and in reality all this massive library is doing is providing the compiler conditions. This all brought me to another thought... Macros run at compile time and more or less write script where you tell it to...some targets support the filesystem package and others don't...that's something a macro could determine...I only want to import the filesystem package for targets that support it...that's an import a macro could write, the same macro that made the determination...

In other words, say hello to my first real usage macro. That's all I really needed. Sure, there is plenty of uphill left with macros but, now it's uphill with momentum. It's awesome. I have only been awake like 36 hours so, I haven't even hit the part where you start having all kinds of free thought and getting messages from god and stuff. I'm already at real-world usage for something much bigger than the macro ~ a more target specific content manager and the elimination of a massive api.

I can be really slow to results sometimes but, supposedly, that and steady wins the race, so....

[MadGypsy]

This post is just a time-killer while I wait to see if an idea I had ever finishes compiling.

In my last post I talked about making my content manager more platform specific and ditching openfl. Well, I went way beyond that. A bunch of my parsers for my content manager weren't really parsers at all, they were just wrapping someone elses parser. This was true for mostly image types. I've banged out animated gifs, png, bmp and tga with true parsers that I wrote. I don't know if any of y'all have ever tried to parse your own png (for instance) but, it's a major pain in the ass. Honestly, IMO, parsing image files is harder than parsing an entire BSP. BSPs are pretty linear. You have a bunch of lumps and they each hold a specific purpose that is unique to the other lumps. Images are nowhere near that simple, especially jpg. I hate jpg. I successfully wrote a writer but it was so hard I completely lost interest in reverse engineering it to be a reader.

uh, well, my thing finally compiled and it looks like there isnt even any errors. Imma go see if my little idea is cool or shitty.

edit: I actually have BSP to thank for these image parsers. Before my BSP parser I had never parsed bytes. ALl my parsers were text or math based. No sooner did I have success parsing BSP and I was already off on parsing mp3 streams. Then I went to wads. I recently did all those image types I mentioned and I think I want to do video streams next.

Woops, I take it back. My first byte parser was .zip but that's sort of a half-ass parser cause one of my api's already had inflate/deflate/etc and it had a class that was easily modified to serve as an Entry so, I'm going to say that it doesn't really count as my first byte parser since I was more like just passing the bytes along. However, that was back in flash days and now I am a Haxe programmer. I absolutely have my own completely-written-by-me zip parser for haxe. I am aware that haxe.zip.Zip exists and works great. I don't care. I didn't write that. I want as much as possible to be MY code. In my build file is a list of APIs I am dependent on. The list isn't very long but my goal is to eliminate that list. I've about halfway done that cause the list used to be longer.

[MadGypsy]

Due to all my recent image parsing I ended up learning a lot about the .gif format. This is a very powerful image format. The grand majority of gifs out there barely tap any of this format's potential.

I had an idea recently to expand the gif format and write a pure JavaScript player for the extended format. This got me thinking about flash. In a broad sense flash is an extended gif played but they made the mistake of linking it to a plug-in. This may have been necessary at the time but, I don't see it being necessary these days.

I can't say that I intend to reinvent flash with automatic browser support but, I have been dicking around in that realm. I used to consider gifs incredibly annoying, mostly due to the fact that support was so shitty. For a long time browsers and whatnot were only properly supporting v87a so v89a gifs would not necessarily play. Also, exporting to a gif was problematic because in many cases it would completely destroy your colors and break your LOD. All that being said, I have a lot more control over gif format now and my ideas/implementations are not at all dependant on the overall support of gif in someone else's product (ie a browser...etc).

I don't know where any of this is going but, I'm enjoying the challenge of shoving my ideas into the current format as much as possible until I have no choice but to extend the format.

[MadGypsy]

idea: expand the bsp lightmap lump

PLIT: save colored lightmap data as indexes to the color palette. replace lightmap lump with "PLIT" + palette indexes

QLIT: convert lit to uint32. replace lightmap lump with "QLIT" + conversion data

This could be implemented in any quake engine/compiler very easily.

[SourceSkyBoxer]

Hello my great AS3 developer ( Haxe developer ) @MadGypsy!!!

Wow I have downloaded your working in process worldspawn bsp from old topic "Real flash with quake??" - Nice work!
Wad is same to Half-Life - wow you save my life because I am lover of Goldsource Engine.
QC is like Query code same to Half-Life for models and sprites.

I am proud to you. You can able wad into AS3. Since I have critic problem. I can not believe you got loading wad.

Great work! PS: I am jealous to you because I start with AS3 but Half-Life members don't believe me - if they say slow for AS3 / Haxe. It is wrong. I have tested your worldspawnbsp ( last version ) like performance if game engine from AS3 is fast and has no problem.

I am very surprised because your work sees better than my starting bsp content of Half-Life. But my project is still not finished. 10 % ops...

I wish I have example code of your. If I know for Half-Life <-> Quake 2/1
Quake 1/2 - Half-Life
wad: Yes - Yes
bsp: 29 - 30, 31 ( Xash3D )
models: ase, obj, mdl2 - mdl
collision of models: Yes - No ( use clip texture )
sprite: pcx or tga - spr
entity data: def, fgd - fgd
map format: map - map-valve220
..

That is different to Quake 2 and Half-Life

[MadGypsy]

Please excuse me if I am wrong but, either your translator or English is a bit hard to decipher, are you saying you are building a flash engine and you need help? If so, I can help you. Don't listen to people about flash. The grand majority of people have no clue what they are talking about and these same people don't have a clue about the current state of the flash player. It out performs neko and (according to all my tests) runs equal to cpp when built from the same framework ( ie HaXe/away3D)

I appreciate your kind words.

I can tell you right now, you should be parsing things into Vector. It is the closest thing you are going to get to a linked list. Also Objects in flash are very fast. So let's suppose you were parsing a wad, you would do something like

var entries:Vector.<Object> = new Vector.<Object>(entry_count)

By telling the vector how long to be in instantiation you can run a .forEach on the Vector and parse all the entries within that loop. I doubt you could do it faster than this without writing code that is very hard to read.

[MadGypsy]

@I wish I have example code of your....

I can give you the entire as3 project. I'd be more than happy to. For me, the pure AS3 version is dead. Not because anything is wrong with it (except it's a bit messy here and there) but because I have chosen to be a HaXe programmer and I have no intention of ever revisiting AS3. I mastered the language long ago and I am even legitimately certified to teach it... time to move on. It's a great language though. I got about 20 years of fun out of AS3 before I moved on to something significantly more complicated.

[MadGypsy]

I took a stab at saving light data as palette indexes. Before I even started I knew one problem would be that light can be any color, including ones not in the palette. Light maps get filtered and that spurred me to try and rewrite the light data in such a way that a conglomeration of colors ( once filtered ) would produce the proper resultant color. As an example, say you had light data that was a 2x2 square and it was supposed to be light blue. Now let's pretend that there was no light blue in the quake palette, only dark blue. My solution would basically checkerboard the square with dark blue and white. It is FAR more complicated than that but, that's a simple gist.

I realize this is basically a useless feature. There is practically nothing to be gained, for the most part. The point behind all of this is really more about skill. Can I pull it off? Would you be able to tell the difference? Those are questions that I would answer optimistically on my worst day.

When I first started this engine my goals were very linear but, I have always been curious... more interested in bringing ideas to life than whatever my intended final goal was. I have already brought so many if my ideas to life in this project that my engine is beyond any more rewrites or backtracks. In other words my code is gigantic and the possibilities I've included already are numerous at even a glance.

One day I will decide that my engine has enough kitchen sinks in it to let y'all play with it, and when that day comes I guarantee you will be blown away.

Recently I had the idea to create what I call a replacement brush. It's a simple idea. Create some brushes that represent things that are far more detailed in your map and properly mark them as replacement. Compile as usual. My engine finds these brushes and replaces them with .obj formatted models. The thing is, collision detection and all that works as expected as long as the mins and maxs of the brush and replacement are basically identical. The .obj can have the light baked on so replacement models don't need secondary uvs. With some really tricky texturing and the right model to pull it off you can use just a sliver of texture and repeat it.... think a hand rail where the shadows are consistent from one end to the other.

It allows you to heavily detail some things and actually end up with less geometry vs how a bsp is cut. I thought about making entire replacement maps but there are issues with that and I need to revisit it after it's boiled in my head a little longer. Mainly a replacement map would be a way for you to replace the entire world but keep the bsp tree for collision purposes.

I have all kinds of ideas. Some of them suck and others don't. I just build them all and let the game developer sort it out.

[MadGypsy]

@SourceSkyBoxer

I wrote a wad3 parser from scratch from you but, I don't have the right palette. If you can hook me up with the proper palette.lmp file I can put the final touches on the parser. I just parsed 3116 miptex images from halflife.wad into images in 6 seconds using pure flash.

The only thing left to do is find wads that utilize all the supported types from the entries struct and write cases for them. If I had the palette and those wads I could put a fork in this in under 30 minutes.

its so simple that I banged it out in one function with 3 loops. I'm only getting the first miptex and the very end is just some nonsense to easily get everything back to the document class. It would be nothing to get all mips and simply erase the end so you can include your own method for retrieving the parsed data.

public static function parse(byName:String, callback:Function):void
{
	var _bytes:ByteArray = Content.wadfile(byName);
	_bytes.endian = Endian.LITTLE_ENDIAN;
	
	if (_bytes.readMultiByte(4, "iso-8859-1") == "WAD3")
	{
		var count:int	= _bytes.readInt();
		var offset:int	= _bytes.readInt();
		
		var entries:Vector.<Object>	= new Vector.<Object>(count);
		var miptex:Vector.<Object>	= new Vector.<Object>(count);
		var images:Vector.<BitmapData>	= new Vector.<BitmapData>(count);
		
		_bytes.position = offset;
		
		//get entries
		entries.forEach( function(val:Object, i:int, self:Vector.<Object>):void {
			self[i] =
			{	position:	_bytes.readInt(),
				dSize:		_bytes.readInt(),
				nSize:		_bytes.readInt(),
				type:		_bytes.readByte(),
				comp:		_bytes.readBoolean(),
				dummy:		_bytes.readShort(),
				name:		_bytes.readMultiByte(16, "iso-8859-1")
			}
		});
		
		//get texture properties
		miptex.forEach( function(val:Object, i:int, self:Vector.<Object>):void {
			_bytes.position = entries[i].position;
			self[i] =
			{	name:	_bytes.readMultiByte(16, "iso-8859-1"),
				width:	_bytes.readInt(),
				height:	_bytes.readInt(),
				offsets:[_bytes.readInt(),_bytes.readInt(),_bytes.readInt(),_bytes.readInt()]
			}
		});
		
		var temp:ByteArray = new ByteArray();
		var iname:Object = new Object();
		
		//get textures
		images.forEach( function(val:BitmapData, i:int, self:Vector.<BitmapData>):void {
			_bytes.position = entries[i].position + miptex[i].offsets[0];
			iname[miptex[i].name] = i;
			
			if (temp.length > 0) temp.clear();
			
			var x:int, y:int
			for (y = 0; y < miptex[i].height; ++y)		//pixels columns
				for (x = 0; x < miptex[i].width; ++x)	//pixels rows
					temp.writeUnsignedInt( Palette.ARGB[ _bytes.readUnsignedByte() ] );
			
			temp.position = 0;				//set pointer at beginning
			
			self[i] = new BitmapData(miptex[i].width, miptex[i].height, true);
			self[i].setPixels(self[i].rect, temp);
		});
		
		Content.set_images(iname, images);
		
		//bullshit way to make it easy
		callback.call({}, {type:"complete"});
	} 
}

[SourceSkyBoxer]

@MadGypsy thanks for explanation!

I am sorry for bad English. I don't use translator. Why they always force me if I am using translator? But I don't use translator. Hello! I have learnt English But my English is not full.

Wow your great code like you have written as halflife.wad reads into AS3.
That is awesome and you're fast.
How do I write ContentManager.as or Content.as
Is it correct or wrong? PS: Sorry I use JPEXS Free Flash Decompiler because I want read about your written code. But I don't want steal. I want check how does it work.

My code looks like this if I don't know how do I getWad or wadFile , static method:

package com.valve.halflife
{
	import deng.fzip.*;
	
	import flash.display.BitmapData;
	import flash.utils.ByteArray;

	public class ContentManager
	{
		private static var _ziplib:FZipLibrary;
		
		private static var _zip:FZip;
		
		private static var _zipfile:FZipFile;
		
		public static var ZIPLIB:Boolean = false;
		
		public static const ZIP_LIST:String = "zip";
		
		public static const IMG_LIST:String = "img";
		
		public function ContentManager()
		{
		}
		
		public static function getWad(byName:String):ByteArray
		{
			if(ZIPLIB)
			{
				if(_zip.exists(byName + ".wad"))
				{
					if(ZIPLIB)
					{
						return _ziplib.getDefinition(byName + ".wad").content;
					}
				}
			}
			
			return null;
		}
		
		public static function getBitmapData(byName:String):BitmapData
		{
			
		}
	}
}

How do I understand if my project loads string of wad like halflife.wad

var test:Boolean = Wad.parse(ContentManager.getWad("halflife");

and I use getBitmapData("AAATRIGGER")

var bm:Bitmap = new Bitmap(ContentManager.getBitmapData("AAATRIGGER"));

How does getBitmapData ( static method of Content / ContentManager ) work?
I am really excited to play with Half-Life under AS3.
If you know how does it work with ExternalInterface with Chromium like Electron or NW.js? I already tried But it is very hard to communicate with ExternalInterface like ApplicatioNDomain or SecurityDomain , Sandbox and I already use NW-Flash-Trust. If you have success with Electron and NW.js than you can tell me how does it work - If it doesn't work than ...

We use Air Captive runtime with BoxedApp Packer or Engima Protector into standalone executable. But bundled executable doesn't work under wine of Linux because I already tried if I bundle whole Air captive application's content into bundled executable and I run bundled air captive application under wine of Ubuntu Guest ( VirtualBox ) and shows error message - "Adobe AIR Installer requires Adobe AIR to be installed."

So sadly! You're right so that we need use Haxe. But I don't understand how do I use haxe.zip api? I wish to write easy zip for OpenFL. Like Nochum.zip or deng.fzip - Did you find other alternative of fzip for Haxe? format.zip or other?

Thanks!