I don't know the details of the history of the first compiler or the first os or how various others were created. But here are some thoughts on how it could be done.
Let's start with a simple computer based on, say, a Motorola 6809 since I'm sorta kinda familiar with that since it is the CPU in my Hero Jr vintage robot. Let's call our computer the Orange.
First how does it run a program at all? Well, on reset, the first thing it does is loads a 16 bit address from the two highest bytes in memory ($FFFE - $FFFF) into the Program Counter. It will then execute the machine instruction at that new address.
So if you could somehow load a program into memory, starting at some address ($D000, say), and also load the address value into $fffe-$ffff the computer will run your program on reset.
One way to do that is to design your new computer with a PROM chip (a "read only memory" that is easy to write to, once) which occupies memory from address $d000-$ffff. That is, when the CPU reads from $fffe-ffff it is reading from the PROM rather than from RAM.
To write your program, you would write it in assembly language and use an assembler to convert to machine code (just a sequence of data bytes). Except we don't have an assembler yet. So let's do it by hand on paper. CPU manufacturers provide info needed to do it by hand.
By the way, we don't have something to program the PROM yet. Let's start with building a machine whose sole purpose is to burn an EEPROM. We need it to have switches on the front (like a PDP8e). You use the switches to enter data 8 bits at a time into any 16 bit address. Let's say it has an 8 switch data input, and 16 switch address input. Also it has Store and Program switches. You can design the hardware with digital logic chips that will do as follows:
- When Store is pressed copy the specified data into the specified RAM address
- When Program is pressed, copy your program from RAM into the eeprom.
- When Run is pressed, run the program starting at the specified address.
I'm leaving out some implementation details for simplicity.
Eventually we want to use our Orange to write, save, and load files, assemble programs, and run machine code generated by assembled programs.
So we need devices for storage (floppy? Tape drive?) and serial UART. Skipping over hardware details, we need software to use the devices. We need to read commands from the terminal, display stuff to the terminal. The commands might include SAVE, LOAD, STORE and RUN. Each command is implemented in software. All this software can go into the PROM and offer these capabilities on reset, similar to 80s home computers like the Commodore 64.
Once we have written, assembled and programmed our PROM to do that, now we have a rudimentary operating system that handles simple input and output. Maybe also add some useful functions that users can call from their own program.
We could make life easier on ourselves and write an assembler and a simple file editor that we can use to write assembly language files. Then we can save the assembly file and run the assembler to create a file of machine code. We can save these new programs to storage. And now we can use those to more quickly write more complex things.
We can add a PROM burner to our list of devices and add the capability to burn programs from memory to a new PROM. We can use that to more conveniently update the PROM on our Orange computer.
It will be easier to write a BASIC interpreter now. Or a compiler. Now you can develop more complex programs quickly. You can rewrite parts of your assembly code in a higher level language so it is easier to maintain.
Maybe you outgrow your computer and want to make an Orange II with a new CPU, more power, memory, etc. well, you already have the first computer so you can develop the ROM code for the new computer using the first computer. You can make a cross assembler and cross compiler. Wire software on the Orange that spits out machine code specific to the Orange II CPU.