SBCL Internals The pages on this CLiki-driven site can be edited by anybody at any time. No warranty of any kind can therefore be made; any implied warranties of merchantability or fitness for a particular purpose are expressly disclaimed

"second genesis", "cold-init" and "warm load"

Krystof:

Let me first explain the difference between "second genesis" and "first genesis", because in many ways it's the most interesting
"first genesis" happens at the end of "host-1", and refers to the dumping of fundamental lispland information in a form that C can understand
it's responsible for creating header files that correspond to the source that has just be compiled
so has correct definitions for tag values, for structure offsets, and so on
apart from that, it's almost completely uninteresting
"second genesis", on the other hand, is tricky
it is responsible for creating a core file that the sbcl binary knows how to load
in the process of doing this, it has to simulate LOADing all of the fasls that have been produced by "host-2"
and, of course, it cheats.
It arranges for function definitions at toplevel to be available very early (see COLD-FSET)
but doesn't do anything about e.g. structures, closures, and other toplevel forms
so at the start of "cold-init" (which is what happens when you run src/runtime/sbcl --core output/cold-sbcl.core)
you have something which knows how to do almost nothing. Just about the only thing it has is an initial function
This initial function needs to be able to create hash tables and packages without doing a full call of anything
because otherwise, if it needs to do a full call, it will need to look up something in a package and will therefore lose
So all the machinery to create packages (and hash-tables, which packages use internally) needs to be inlineable
So far, so theoretical.
hash-tables are structures; structures use SYMBOLICATE, which your "(concatenate 'string (the simple-string (string (car things))))" comes from
(if I remember correctly)
so we must avoid a full call to CONCATENATE because we can't do full calls yet.
OK, let me (for posterity) explain the rest of cold-init
cold-init (which, I remind the audience, is what happens when we boot our fresh cold-sbcl.core)
punts us into an initial function (called #'cold-init, probably, but I forget)
which first sets up random, then hash-tables, then packages.
Then starts setting up various other basic bits of the system, including the type system (see src/code/cold-init for the precise order)
and then runs all the stored toplevel forms
these toplevel forms are those that weren't dealt with by "second genesis": i.e. everything that isn't a COLD-FSET at toplevel
everything that would have happened had one done LOAD on the lisp-obj file, basically
so things like defining various structures, setting up the compiler globaldb, etc etc
once it's done all the toplevel forms there are a few remaining fixups to do
setting up compiler policy, calling various OS procedures
and then it's into "warm load" aka "try (* 2 3) and see if it works"
warm load is essentially just "compile these files that we haven't figured out how to put into cold-init" which in practice means "PCL and those files that depend on it"

cliini:

Apparently you just dump the image at the end of cold-init?

Krystof:

No. We dump a fake image at the end of "second genesis"
we run src/compiler/generic/genesis as an application in the host lisp
it effectively loads in all the fasls in obj/from-xc/*
and fakes up a core (called cold-sbcl.core)
based on what those fasls contain
with an initial function that will execute the fixups I've been describing
"cold-init" is the act of running those fixups, and is usually where things break
("cold-init" is where the first lisp-as-target-code is actually executed)

cliini:

Um. Does it run those fixups on the core FILE?
I thought it was doing this in memory.

Krystof:

Sorry, yes, it's doing this in memory
the core file is mmap()ed

cliini:

So then it does need to output something at the end of cold-init to make the actions persistent.

Krystof:

Oh, sorry. Yes.
What happens next is that _after_ all of these "cold-init" fixups are run, we effectively have a working compiler
so we don't bother to save and reload, but launch immediately into "warm-init"
which essentially involves compiling PCL

cliini:

And the end of warm-init dumps the final core?

Krystof:

and then at the end of warm-init we save a core. And that is the end-user sbcl.core

cliini:

OK, now I think I understand.

!cold-init in detail

The function !COLD-INIT is called by the SBCL runtime at the beginning of life in lisp on the target. When first loaded cold-init can really just call functions--genesis has statically linked up function calls to work. Other aspects of the system are not yet wired up as described to some degree above.

(To really document cold init we need to explain second genesis in detail, which we only do roughly at the moment. i.e. explain how layouts, strings, packages, numbers, etc. are loaded from FASLs and then how they are represented in a core file.) Briefly, genesis tries to cross-compile as much as it sanely can--functions, symbols, strings, constants--look at the cold FOPs in genesis to get the dirty details of how cold-load works. (And then fill in the wiki.)

Whatever the details of the COLD-INIT environment, what follows is an ordered list of things that cold init sets up:

1. Inhibit garbage collection
2. !eval-cold-init sets some parameters for eval
3. thread-init-or-reinit creates a thread object for the initial thread and a few other objects/sets up threading specials.
4. !random-cold-init initializes the *random-state* structure
5. !hairy-data-vector-reffer-init (don't know exactly what this does
6. !character-database-cold-init and !character-name-database-cold-init set up the character database for unicode and other formats
7. !early-package-cold-init and !package-cold-init create packages from the list of of *!initial-symbols* cold-loaded into the core by genesis
8. !globaldb-cold-init initializes the info? system--a the "global database" that provides "a functional interface to globa information about named things." Seems to copy from the globaldb loaded on the SBCL compiler from host via read macros.
9. !function-names-cold-init and !fdefn-cold-init sets the :function :definition info for each fdefn. Assisted by genesis, which filled in the value *!initial-fdefn-objs*
10. !type-class-cold-init, !typedefs-cold-init, and others set up the type system
11. !policy-cold-init-or-resanify
12. !constantp-cold-init and !early-proclaim-cold-init
13. top-level forms that were collected by the cold FOP machinery and cold-loaded into the cold core are now executed. Apparently this is kind of kludgy because so-called "fixups" are intermixed with top-level forms.
14. this list is incomplete (see below)

I (Red Daly) did not finish documenting cold-init. below is the remainder of the !COLD-INIT function that remains to be explained:

  (show-and-call !policy-cold-init-or-resanify)

  ;; Only do this after toplevel forms have run, 'cause that's where                                                                                                                                                                         
  ;; DEFTYPEs are.                                                                                                                                                                                                                           
  (setf *type-system-initialized* t)

  ;; now that the type system is definitely initialized, fixup UNKNOWN                                                                                                                                                                       
  ;; types that have crept in.                                                                                                                                                                                                               
  (show-and-call !fixup-type-cold-init)
  ;; run the PROCLAIMs.                                                                                                                                                                                                                      
  (show-and-call !late-proclaim-cold-init)

  (show-and-call os-cold-init-or-reinit)
  (show-and-call !pathname-cold-init)
  (show-and-call !debug-info-cold-init)

  (show-and-call stream-cold-init-or-reset)
  (show-and-call !loader-cold-init)
  (show-and-call !foreign-cold-init)
  #!-win32 (show-and-call signal-cold-init-or-reinit)
  (/show0 "enabling internal errors")
  (setf (sb!alien:extern-alien "internal_errors_enabled" boolean) t)

  (show-and-call float-cold-init-or-reinit)

  (show-and-call !class-finalize)

  ;; The reader and printer are initialized very late, so that they                                                                                                                                                                          
  ;; can do hairy things like invoking the compiler as part of their                                                                                                                                                                         
  ;; initialization.                                                                                                                                                                                                                         
  (let ((*readtable* (make-readtable)))
    (show-and-call !reader-cold-init)
    (show-and-call !sharpm-cold-init)
    (show-and-call !backq-cold-init)
    ;; The *STANDARD-READTABLE* is assigned at last because the above                                                                                                                                                                        
    ;; functions would operate on the standard readtable otherwise---                                                                                                                                                                        
    ;; which would result in an error.                                                                                                                                                                                                       
    (setf *standard-readtable* *readtable*))
  (setf *readtable* (copy-readtable *standard-readtable*))
  (setf sb!debug:*debug-readtable* (copy-readtable *standard-readtable*))
  (sb!pretty:!pprint-cold-init)

  ;; the ANSI-specified initial value of *PACKAGE*                                                                                                                                                                                           
  (setf *package* (find-package "COMMON-LISP-USER"))

  (/show0 "done initializing, setting *COLD-INIT-COMPLETE-P*")
  (setf *cold-init-complete-p* t)

  ; hppa heap is segmented, lisp and c uses a stub to call eachother                                                                                                                                                                         
  #!+hpux (sb!sys:%primitive sb!vm::setup-return-from-lisp-stub)
  ;; The system is finally ready for GC.                                                                                                                                                                                                     
  (/show0 "enabling GC")
  (setq *gc-inhibit* nil)
  (/show0 "doing first GC")
  (gc :full t)
  (/show0 "back from first GC")

  ;; The show is on.                                                                                                                                                                                                                         
  (terpri)
  (/show0 "going into toplevel loop")
  (handling-end-of-the-world
    (toplevel-init)
    (critically-unreachable "after TOPLEVEL-INIT")))

This page is linked from: Basics   Build   genesis   Stop and Copy GC   warm initialization  

CLiki pages can be edited by anyone at any time. Imagine a fearsomely comprehensive disclaimer of liability. Now fear, comprehensively