| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This lexer is being written as a chunklet, not quite quite complete yet
as it lacks some functionality to make it generally useful for things,
but good enough for the gluegen use case now. So it's in the repo now
and we can go ahead and use it for this instead of having this
hacked-to-pieces third party thing that's nowhere near as efficient. In
the future the goal is to have a decently usable library for any sort of
C metaprogramming needs, not just within this project.
This design does the data-oriented thing of storing as little about each
token as possible (just 5 bytes), and re-lexing specific pieces only
when necessary. In some cases, full validation of correct syntax is
only possible through this secondary step. Since the use case is
metaprogramming and code-generation rather than the reimplementation of
Clang, this slight sloppiness in validation doesn't seem too bad.
It's been a while since I did a proper performance measurement of this
code but in some crude tests I did in the past the primary tokenisation
step was running at well over 500MB/s, which is fast enough for me. It's
possible that this version is a tiny bit slower due to the added
complexity of making UCNs work. UCNs, incidentally, are one of the
dumbest features of C by far. However, unlike trigraphs - which this
lexer does not handle - UCNs are still in the language, so it's kind of
sort of necessary to support them.
It's probably still possible to come up with a faster design with some
SIMD trickery, but the main loop is currently branchless and the lookup
tables are too big for the SIMD lookup things, so that seems kind of
hard. A separate SIMD path for whitespace or comment runs seems dubious
as it would introduce branch mispredictions everywhere. I also made
previous attempts to unroll the main loop and every attempt just made it
slower, so I guess code size is a significant factor.
Optimising the secondary tokenisation of identifiers (and later numerals
and string/character literals once those are handled) is still on the
cards, but since that happens less often, I don't know how much
difference it'll make.
At any rate, in a multithreaded context this thing would already come
pretty close to SSD speeds, if open-read-close syscall overhead doesn't
get in the way first. I imagine it's fast enough for anyone who hasn't
*already* written something faster.
|
| |
|
|
|
|
|
|
| |
This is for somewhat later. I'd always planned to use it - it existed
already in earlier private repos, in fact. I just didn't bother to
import it here in case it wouldn't actually be needed, but with the way
current plans are going, it's definitely going to be needed, so here it
is.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
hook_inline() uses the new x86_len() function to get instruction lengths
instead of doing full-blown disassembly, which should be a tiny bit
quicker, and also removes the next for about 90KiB of lookup tables and
such in the final binary. The code-digging logic in demorecord is also
rewritten to be opcode-based rather than mnenmonic based. In general,
going forward the plan is to always rely on opcodes and thus avoid a
bunch of disassembly work every plugin load.
udis86 is still in the tree for now to provide dbg_asmdump(), but it's
only compiled into debug builds and left out of releases completely. As
such, the whole BSD licence statement is also gone from the distribution
LICENCE files. There's now also a dbg_toghidra() which spits out a
rebased address to look stuff up for proper reverse engineering, which
might be more useful than dbg_asmdump() anyway. If nobody ends up using
the latter ever again, udis86 could get chucked completely. We'll see.
Also shoehorned into this commit are a couple more forgotten copyright
year bumps and some general minor cleanup here and there, because I
couldn't be bothered wading through all the diff hunks. Oh, and
makebindist.bat now makes an effort to make the zip file timestamps
predictable/reproducible. That should be a different commit for sure,
but oh well too bad.
|
| |
|
|
|
|
|
| |
I had plans to use these at some point but never did. Maybe I still will
later but listing them otherwise is a bit confusing and weird, and stems
from an oversight when I initially imported all the early dev stuff into
the public Git tree.
|
|
|
With code from Bill. Thanks Bill!
|
Michael Smith
