The line in the log

The request that broke the run was a 404 for a path I do not serve, with a query string that read like a fragment of a PHP inclusion probe: %ADd+allow_url_include=1. Three printable characters stood where a byte belongs, because the sender percent-encoded it. My log format uses the default escaping, which rewrites quotes, backslashes, and any byte below 32 or above 126 as \xXX 1. That rule has no reason to touch %AD, so the line landed as plain ASCII and every tool downstream treated it as text.

That part is ordinary. Every request line in an access log is a stranger’s string, and the line above is a 404 nobody needed to answer. My own code made the byte matter.

The byte comes back out

Percent-encoding is a transport convenience, and something has to undo it. In my chain the decoder is GoAccess. It unescapes a log line’s request and query fields through decode_hex in util.c, called from decode_url in parser.c, which reads two hex digits and writes the named byte into a character buffer 2. A value above 0x7f goes in with no UTF-8 check on the way.

The JSON writer in the current source does check. escape_json_other measures each sequence with utf8_seq_len() and writes \ufffd for an ill-formed one, and its own comment gives the reason: invalid UTF-8 “would make the whole document invalid JSON” 3. The file my generator read had a raw 0xad in it. I could spend an evening working out which build of which component skipped that guard and learn a version string for my trouble. The artifact on disk was not valid UTF-8, and surviving that was my read path’s job.

RFC 8259 is blunt about the writer side: JSON text exchanged between systems that are not part of a closed ecosystem “MUST be encoded using UTF-8” 4. A MUST in a spec describes what a writer owes a reader. It says nothing about the file on your disk.

One bad byte, every hour

My generator opened all.json with a plain read and json.load. Python’s default error handler for decoding is strict, which raises UnicodeDecodeError at the first byte that does not fit the encoding 5. Nothing in my script caught that exception, so the process died mid-run.

The damage went past one missing hour. The script printed its output into summary.txt in place, so the traceback overwrote the last good digest and its numbers. Every hourly run after that read the same all.json and died the same way. My morning check found an error string where traffic numbers used to be.

I built a one-line test with the same byte in it before I changed anything:

>>> b'{"req": "GET /?%ADd+allow_url_include=1"}\xad'.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 41: invalid start byte
>>> b'{"req": "GET /?%ADd+allow_url_include=1"}\xad'.decode('utf-8', errors='replace')
'{"req": "GET /?%ADd+allow_url_include=1"}�'

The second read returns the document with one character swapped for U+FFFD, the replacement marker 5. The counts in that document stay readable, so the digest runs on a file that would have stopped it before.

The boundary I moved

I made three changes to the generator.

The read passes errors="replace" now. The bad byte becomes U+FFFD, the rest of the document parses, and the numbers compute. A scanner’s probe decides nothing about whether I get a report.

Output goes to a temp file that renames into place only after a successful run, so a failed hour leaves the last good hour’s digest untouched. That trade puts a stale digest at the same path as a fresh one, and the filename gives no sign that an hour failed. The build time inside the file is the only thing that separates them.

Failures go to the unit journal instead of into the report. The digest carries its own build time and I read that line before I trust a number, because the freshness check belongs in front of the reader rather than in a log I open after something already looks wrong.

The parser sits in the attack surface

Every string in an access log that my server did not write comes from whoever sent the request: the path, the query string, the user agent, the referrer. OWASP’s logging guidance treats event data from another trust zone as untrusted, because it may be missing, modified, forged, replayed and could be malicious 6. The same page lists availability as a logging risk and names the failure mode I hit: an attacker uses one log entry to destroy other log entries 6. One line cost me a day of traffic numbers.

I cannot stop a stranger from sending a percent-encoded byte. I can stop my read path from treating a file as trustworthy because a tool I run wrote it. The test costs one fixture, a synthetic access log line with %AD in the query string, piped through every step, with a written digest as the pass condition rather than a clean exit code. I run it after any change to a step in that chain, and it runs before I read the morning numbers.

The next hourly run wrote a full digest with the byte replaced and the counts intact. The byte stays in the fixture.


  1. Nginx. (n.d.). Module ngx_http_log_module. https://nginx.org/en/docs/http/ngx_http_log_module.html ↩︎

  2. Orellana, G. (2026). goaccess source: src/util.c and src/parser.c [Source code]. GitHub. https://raw.githubusercontent.com/allinurl/goaccess/master/src/util.c and https://raw.githubusercontent.com/allinurl/goaccess/master/src/parser.c ↩︎

  3. Orellana, G. (2026). goaccess source: src/json.c [Source code]. GitHub. https://raw.githubusercontent.com/allinurl/goaccess/master/src/json.c ↩︎

  4. Bray, T. (Ed.). (2017). The JavaScript Object Notation (JSON) data interchange format (RFC 8259). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc8259.html ↩︎

  5. Python Software Foundation. (n.d.). codecs: Codec registry and base classes. Python 3 documentation. https://docs.python.org/3/library/codecs.html ↩︎ ↩︎

  6. OWASP Foundation. (n.d.). Logging cheat sheet. OWASP Cheat Sheet Series. https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html ↩︎ ↩︎