Theoretical & Applied Problems


Most-Used Commands[§V.B]

This is an up-to-date ranking of my most-used Linux commands...


...on the machine hosting this site:

...on my personal laptop:

  1. cd: 96
  2. ls: 86
  3. vim: 36
  4. mocp: 34
  5. python3: 25
  6. sudo: 21
  7. man: 16
  8. history: 16
  9. find: 15
  10. update-alternatives: 14
  11. git: 14

Table of Topics

You'll notice, these are loosely sorted from most abstract to most practical.

As a consequence, the most impressive ideas are at the top, but each layer of abstraction is fascinating in its own right.


  1. Peace, Love, and Math — That Which Makes Life Worth Living
    1. The Best of Discrete & Recreational Mathematics

  2. (Ever-so-slightly) Applied Math
    1. Cybersecurity Articles
    2. Beautiful Algorithms

  3. How To Become a Polyglot
    1. The C++ Dialogues
    2. Less Enjoyable Languages
      1. Python
      2. JavaScript
      3. C#

  4. Tech Support Archive
    1. Recommended FOSS Software
    2. How I Broke My Computer This Week
    3. An Impossible Challenge
      1. Server-side
      2. Client-side

  5. Specific Platforms
    1. Obscure Linux Distros I Want to Try
    2. Favorite Unix Commands
    3. FFmpeg Flags

  6. Ranting & Venting: The Search for a Good Job
    1. How They Locked Me Out of Interesting Jobs This Month
    2. The Wall of Shame: Featuring Companies that Ignored Me

At some point in his or her life every working mathematician has to explain to someone, usually a relative, that mathematics is hardly a finished project. The mathematicians know, of course, that it is far too early to put the glorious achievements of their trade into a big museum and become happy curators. Our subject has, in certain respects, hardly begun.
— Barry Mazur, Foreword to Fearless Symmetry

The Best of Discrete & Recreational Maths

This is a work in progress! It will showcase some of the best, most underrated theorems in maths. Stay tuned and wait for your mind to be blown — or if you're impatient, solve the math jester challenges and submit your solutions to see the deeper meaning behind each problem.

Geometry

Linear Algebra

Recreational Maths



'Tis not too late to seek a newer world.
Push constants to the stack machines,
And sitting well in order smite
The sounding furrows; for my purpose holds
To sail beyond the VM stacks, and the baths
Of all the western stars, until I die.
It may be that the GOOG will wash us down;
It may be we shall touch the Hasty Likes,
And see the great Turing, whom we knew.
Though much is taken, much abides; and tho'
We are not now that strength which in old days
Moved earth and heaven, that which we are, we are,
One equal temper of heroic hearts,
Made weak by time and fate, but strong in will
To strive, to seek, to find,
and not to yield [control].
— Based on "Ulysses" by Alfred Lord Tennyson (1809-1892)
Modded by FX of Phenoelit; republished in Phrack Magazine

Cybersecurity Articles

Code Exploits

How to Know When You're in a Simulation

How to Know If Your Phone is Secure (Hint: It's Not)

Every year, nay, every month, there are new security vulnerabilities being found and exploited. This is especially true viz-à-viz our phones. Here I intend to keep a consolidated list of security vulnerabilities related to people's phones — security holes that most people are blissfully unaware of.

Cellular data plans got more affordable during recent years and cellular network coverage increased. Disabling Wi-Fi by default and only enabling it when using trusted networks can be considered a good security practice, even if cumbersome.

The paper's mitigation recommendation includes "disabling Wi-Fi by default," which suggests that we should be using mobile network data plans by default instead. However, using these networks has drawbacks and security vulnerabilities of its own:

There are even modern exploits that use the phone's sensors, including the gyroscope, to collect data on the phone's user. Take a look at those articles, and Dr. Guri's site for more.


Stay secure, my friends.



Computer science is no more about computers than astronomy is about telescopes.
— Edsger Dijkstra
Corollary: The best computer science courses are the ones that don't ever require you to touch a computer.
— The Math Jester

Fancy Algorithms

This is a place to collect interesting algorithms, many of which I did not even learn in my university curricula. Join me on a tour of some eye-opening algorithms. These are genuinely impressive creative accomplishments.



Never trust a programmer who says he knows C++.
Louis Brandy

The C++ Dialogues

This is a collection of dialogues, some real and others imaginary, which help to solidify knowledge of C++. If you're reading this, please treat these as approachable lessons and parables, inspired by those in The Number Devil, Fearless Symmetry, and other books. The best part is, the code provided is easily copiable — so you can verify the results on your machine!

Prerequisite: These parables are geared toward someone with knowledge of C.

Disclaimer: They are provided as a reference[1] with NO WARRANTY.

I intend to use my own Dewey-Decimal System for organizing these... It will look something like the following:

0xxx These are supposed to be high-level, less technical questions that a C++ beginner might have. If you want to start with something less technical, then start with this section.
1xxx These are supposed to be basic technical questions about some of the perceived paradoxes of C++. The author posts them here as an opportunity to share what he wishes he had known when starting C++.
2xxx A section dedicated to imaginary dialogues about intermediate concepts.
3xxx These will include lessons deemed to be more advanced, e.g. copy elision and relocatable objects.
4xxx This is the Restricted Section of the library. (lol)

Seriously though, this will contain some questions and comments about the Assembly code to which C++ compiles.

All hail the almghty compilation pipeline; it is beautiful.
5xxx These dialogues are inspired by (or outright excerpted from!) conversations with the experts on IRC. Relative to the other dialogues, they are expected to be of the utmost quality and the highest difficulty.


[1] no pun intended.



The Language Rants

Python — The Twitter of Programming Languages

  1. A significant fraction, between 11 and 46 percent, of packages distributed on PyPI have security vulnerabilities.

  2. Whitespace is required. No language should require a particular quantity of blank space. Rather, the interpreter should be more concerned about semantics than about forcing a certain formatting scheme. More specifically, curly brackets should always be an option for the programmer to use to specify the bounds of a control structure.

  3. Variables do not have a scope. You can have a variable that is defined within a nested control structure and use it outside that control structure by accident (e.g., by merely copying and pasting code and forgetting to rename a variable reference)! This has happened to me before, and it is insane.

  4. Common English words, like sum, min, and list, are built-in functions in CPython, so it is a bad practice to use them as identifiers. This may not sound like such a bad thing — after all, it incentivizes you to "just use more descriptive names" — but it also restricts the developer's ability to write code that (a) is good-quality (using distinct variable names) and (b) doesn't take too long to write, e.g. for Advent of Code challenges, etc. It also appears to be an inescapable feature (is it not?).

  5. People give you a weird look for using semicolons to demarcate the end of a statement.

  6. Error messages are so often counter-intuitive and unclear. Allow me to present some examples.

File "/home/chozorho/ctf/advent/2021/11/./solve.py", line 77, in <module>
if (newBoardState[size-1][width-1] > 9):
TypeError: 'int' object is not subscriptable

This message is infuriatingly vague. It uses one and only one line to describe the error, and the description uses no more than six words. It does not state which expression corresponds to the 'int' object (e.g., is it newBoardState, or is it newBoardState[size-1]? etc.). It also does not give any column index, so this information cannot be easily inferred. In fact, because python is dynamically typed, it is difficult enough to track down the error: whereas a compiler would have checked this ahead of time before blindly running the program, with errors like these, the programmer must waste time re-checking the code manually. In this case, the error was caused on an earlier line of code, so as it turns out, the line being printed isn't even that helpful.

Traceback (most recent call last):
File "/home/chozorho/git/redacted/censored_main.py", line 71, in <module>
censored_locations = inner_object.get_stuff_locations(param1, empty_list)
File "/home/chozorho/git/redacted/censored/inner_object.py", line 54, in get_stuff_locations
loc_div = selenium_driver.find_elements_by_class_name(censored_html_class_name)
TypeError: 'NoneType' object is not callable

How fun, another six-word message. The documentation suggests that find_elements_by_class_name is a valid method on a Selenium webdriver. In practice, it does not exist; no problem, it is probably a simple matter of different versions in use. But then, why should the error mention a "NoneType object" if the selenium_driver object is not a NoneType?? If it is a method that is not found, then this should be made as clear and consistent as possible, without using any ambiguous or counter-intuitive language about "NoneType objects." Anything else runs a high risk of communicating poorly and acting smug when you're misunderstood:

Traceback (most recent call last):
File "/home/chozorho/git/redacted/censored_main.py", line 71, in <module>
censored_locations = inner_object.get_stuffs_locations(param1, empty_list)
File "/home/chozorho/git/redacted/censored/inner_object.py", line 66, in get_stuffs_locations
startInd = loc.index(">") + 1
File "/usr/lib/python3/dist-packages/bs4/element.py", line 1374, in index
raise ValueError("Tag.index: element not in tag")
ValueError: Tag.index: element not in tag

Does python have a rule that their error messages cannot exceed six words??? Apparently, even though I am printing the value of loc and see it displayed as a string, I soon discover that the variable loc is not even a string. This is yet another perfect example of how dynamic typing causes headaches and confusion at runtime.


JavaScript — "wat?"

The auto-casting rules in JavaScript always confused me. Let me reproduce a table to explain why.

Expression Numerical Value String Value Boolean Value
false 0 "false" false
true 1 "true" true
0 0 "0" false
1 1 "1" true
"0" 0 "0" true
"000" 0 "000" true
"1" 1 "1" true
NaN NaN "NaN" false
Infinity Infinity "Infinity" true
[] 0 "" true
[20] 20 "20" true
null 0 "null" false
undefined NaN "undefined" false
"false" NaN "false" true

The reason why this is so confusing is partly because it is not self-consistent: if you pass a variable through multiple stages of conversion, its boolean value may change in unexpected ways. If you read a string from the user, and they type "false", then casting that string to a boolean yields true, but casting it to a number and THEN casting to a boolean yields false. Casting [3, 14] to a number and THEN to a boolean yields false, but converting it directly to a boolean yeilds true. See what I mean?

But it gets even worse.

When you are working with an array, and you try to retrieve an element outside its bounds, you get an undefined. Simple enough, right? But here's the problem: you can manually set a value in the array to undefined (in which case the array length doesn't change!), and you can even set a value of the array at an out-of-bounds index! This is absurd because it violates the very concept of a fixed-length array.

But it gets even worse.

It turns out that, much like PyPI packages have security vuilnerabilities, so do the packages on NodeJS.

This reinforces a joke I have made in the past: that Python and JavaScript are in constant competition for the title "Least Enjoyable Programming Language."

C# — "That Friend of a Friend"

Every time I have this discussion with a C# fanboy it always goes like this:

A: Yea, use C#!! It's faster and better-designed than crappy old Java!
B: But I run Linux. I refuse to accept spyware and keyloggers on my computer just to be able to use a goddamn programming language.
A: Aw, don't worry root! C# is cross-platform! It's great! You can just download this DotNET Core on Linux and voilà!
B: Huh, ok, maybe I'll try it out.
...
B: Oh, by the way, does your C# game project run on Linux?
A: What? Oh no of course not lol. It uses <insert library that isn't supported on Linux, e.g. winforms>, silly! 🤪
B: Count me out.



Selected FOSS applications

Here are some obscure protocols & FOSS applications, if you're into that sort of thing:

This is something I'm still wrapping my head around. Apparently the vtools, which I finally found on another blog post, is an implementation of a "republican version control system." I don't understand what this means yet, but it seems interestng...

This is a program that will allow you to access an Android phone from your laptop. After a lot of struggling and debugging, I have successfully tested this program from my Linux machine.

I also want to save this bookmark about restarting the ADB server — which addresses an error I frequently got when running scrcpy.

To access my phone and send texts from my computer has been a dream of mine for years (I will make a blog post about this sooner or later). Now this dream is becoming a reality!



How I Managed to Break My OS This Week

99 little bugs in the code.
99 little bugs in the code.
Take one down, patch it around,
127 little bug in the code...
— The Programmer's Drinking Song

This is a place to collect many technical problems I've faced, across my different Linux machines (laptop and Linux server). Treat it as a way to store some valuable life lessons, and share them with whoever stumbles upon this page. A common theme you'll notice is that getting help from people on IRC makes solving these problems so much easier.


Episode I: Devuan Linux Breaks Again!


#!/usr/bin/env bash
set -o xtrace

# step 0: set non-root user settings
mkdir /home/devuan/misc /home/devuan/.ssh
echo "set nu hlsearch" | tee /home/devuan/.vimrc

# step 1: decrypt and mount the disk drive
sudo cryptsetup open --type luks /dev/sda3 crypto_LUKS
sudo mount /dev/quicksilver-vg/root /mnt
sudo mount /dev/quicksilver-vg/home /mnt/home
sudo swapon /dev/quicksilver-vg/swap_1

# step 2a: copy SSH key
cp -pR /mnt/home/chozorho/.ssh/id_tmj_cho_rsa /home/devuan/.ssh/id_rsa

# step 2b: download the firefox developer edition tarball itself
cd ~/misc
wget https://www.themathjester.com/common/firefox-93.0b8.tar.bz2

# step 3a: initiate firefox developer edition to populate the local profile
tar -xjvf firefox-93.0b8.tar.bz2
cd firefox/
pushd .
#local_ffdev_pid=`./firefox & echo $!`
#local_ffdev_pid=`sh -c "echo $$; exec ./firefox"` # TODO debug this launch!!
#sleep 6

# step 3b: auto-kill this browser instance...
#kill -9 $local_ffdev_pid
# This automatic "kill -9" feature is pending testing/optimization...
# For now, simply close the browser manually when it opens.
./firefox

# step 4: get the new, temporary (livecd local) profile name, e.g. do0o40b9.dev-edition-default
cd ~/.mozilla/firefox
local_profile=`ls -t | grep dev-edition | head -n 1`

# step 5: get the inner profile name and copy it. DO NOT MODIFY IT!!!
inner_profile=`ls -t /mnt/home/chozorho/.mozilla/firefox | grep dev-edition | head -n 1`

# step 6: actually run the replacement routine. Delete and copy the new into the old.
rm -rf $local_profile
cp -ipR /mnt/home/chozorho/.mozilla/firefox/$inner_profile $local_profile # do0o40b9.dev-edition-default/

# step 7: re-run firefox. The correct profile should be selected!
popd
./firefox &

# step 8: install preferred software. Sit back and relax while it gets installed!
sudo apt-get install moc newsboat build-essential git git-core texmaker \
valgrind fonts-arphic-ukai fonts-arphic-uming \
fonts-ipafont-mincho fonts-ipafont-gothic \
fonts-unfonts-core


How I Fixed My OS

As it turns out, this is caused by an upgrade to GCC 10 and libpthread.

The thread above claims that "the upload of gcc-10 moved libgcc_s.so to /lib instead of /lib/x86_64-linux-gnu." But, on my machine, the library is in /lib/x86_64-linux-gnu and not in /lib. I try copying the library to make sure it is in both locations, but that doesn't fix the issue. Am I missing something obvious? It seems I need to modify a find command somewhere, but it's not clear where to add this.

In October, I finally find a time to ask the gods on IRC for some help. And sure enough, they are able to give me various recommendations for solving the root of the problem. I chroot into the system, mount the /boot files, and then re-install the initramfs for kernel version 5.10.0-8. I also may have re-installed grub2, but I cannot recall if that made a difference.

On October 15th, after doing another apt-get update/upgrade to be safe, I confirm that I can decrypt my drive and log in as usual. So: the issue is finally resolved!


Episode II: The Agony of Trying to Connect (to a Database)

Am I alone in thinking that the hardest part of backend webdev is setting up and connecting to a database in PHP?

Ever since I first tried to learn PHP (which was over a decade ago, and I had no idea what I was doing), and proceeding to this day, I have had endless amounts of stress and confusion whenever I try to glue the PHP and database service together my own.

Let me illustrate the most recent issue I've had — and mind you, this is just one small taste of the PHP difficulties I've experienced.

I decide to actually try out a NoSQL database for a change. Therefore, I try to install mongodb on my Linux host (in fact, it is the same Debian VM hosting this website! Say hello).

Thankfully, I am able to install this fairly easily; the instructions on the MongoDB website are very intuitive and to-the-point, and apt installs version 5.0.5.

I soon stumble upon some demos from tutorialspoint. Aha, I think to myself, this looks promising. I've used tutorialspoint before, in my efforts to learn perl and Rust (these deserve their own articles!) and when researching things like AJAX and Selenium. I know they won't let me down this time!

Alas, the very first line of their PHP script results in a failure. I dig deeper into the Apache logs (which is already a slightly painful endeavor, since I must invoke root permissions to access them) and find the following error:

[Thu Jan 13 05:46:44.933952 2022] [php7:error] [pid 23821] [client **.**.**.**:53632] PHP Fatal error: Uncaught Error: Class 'MongoDB' not found in /var/www/html/*****.php:31\nStack trace:\n#0 {main}\n thrown in /var/www/html/*****.php on line 31

For better or for worse, I am able to find a StackOverflow post that matches this error message (see also, my blog post criticizing of StackOverflow). This one at least cites a source that also suggests instantiating the MongoDB\Driver\Manager.

But as soon as I go to run this recommended one-line adjustment, I see a awfully simiar-looking error:

[Thu Jan 13 08:23:15.791746 2022] [php7:error] [pid 23821] [client **.**.**.**:53632] PHP Fatal error: Uncaught Error: Class 'MongoDB\\Driver\\Manager' not found in /var/www/html/*****.php:31\nStack trace:\n#0 {main}\n thrown in /var/www/html/*****.php on line 31

How does this make any sense??? Have I really not installed the right PHP MongoDB driver? But I had already installed php7.4-mongodb, which matches my active version of PHP (that's right, active version... more ranting about that is sure to come later) and thus should've done the job! After all, when I invoke ls /usr/lib/php/20190902/ as recommended in still other StackOverflow posts, I indeed see a mongo.so in my installed extensions.

I proceed to look up this latest error on DuckDuckGo, yet the top search results are different errors (e.g., connection timeout, authentication failure, an error in the pthreads extension, etc.). No luck.

Do I seriously need to install this specific MongoDB PHP Driver using PECL? If so, why isn't it enough to do the php7.4-mongodb installation above? Only time will tell; I am too exhausted to continue.


Intermission

To this day, part of me wonders: if it is the case that this is really just me, and no one else has trouble setting up databases and connecting with PHP, then perhaps webdev just isn't my forte.

What do you think? Am I a worthless comp sci major? Or am I just exhausted due to a lack of sleep?


Sure enough, the following night, I try to install the aforementioned "mongo-php-driver." The command recommended on that GitHub page fails, with a simple "pecl: command not found" error. Ok, so I need to install PECL somehow.

I stumble through the top search results, which are vague websites stating that pecl is accessed through PEAR, which can be installed with a simple php invocation. But does this really help me? No. While it does give me access to pearl and pecl executables, once I try to actually use them to install mongodb (you know... like the GitHub page suggests), they exit with the following error:

WARNING: channel "pecl.php.net" has updated its protocols, use "pecl channel-update pecl.php.net" to update
downloading mongodb-1.12.0.tgz ...
Starting to download mongodb-1.12.0.tgz (1,392,375 bytes)
.........................................................
...done: 1,392,375 bytes
633 source files, building
running: phpize
sh: 1: phpize: not found
ERROR: `phpize' failed

This is becoming increasingly frustrating the further I go. I then attempt to do research on this new command phpize.

Upon doing research on phpize, I need to remember to disregard the search result titled "How to install phpize for PHP7+", which, I kid you not, literally redirects to the base domain site newbedev.com. From other search results, I discern that it is used in the build-process for PHP extensions, which at least fits into the knowledge I already have, and it is allegedly included in a package called php<version>-dev. I then try to install that package as root, which miraculously works (all hail the debian maintainers).

Then I can finally run pecl to install the full, unmitigated mongodb package, right? The culmination of all this work! To the victor go the spoils!

...

The build proceeds...

...

...and terminates with the following few lines:

Build process completed successfully
Installing '/usr/lib/php/20190902/mongodb.so'
install ok: channel://pecl.php.net/mongodb-1.12.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=mongodb.so" to php.ini
Segmentation fault

This is extraodinarily frustrating. Running a package manager ends in a segfault?? Normally, I am happy to debug software using GDB to diagnose a problem, but this is a package manager! One would expect it to be well-maintained and robust.

The only saving grace is that the error message gives a clear prescription for how to avert the problem (that is, assuming that the "you should add" suggestion is even related to the segmentation fault, which is no guarantee). But here's the problem: I have been checking my PHP configuration files (/etc/php/7.4/cli/php.ini and subdirectories) numerous times, and I have triple-checked the mongodb.so is set there explicitly.

None of this makes any sense.

How I Fixed My OS

As always, asking for help on IRC reveals answers that you never would've thought possible (see also, A Word with the Gods, a C++ dialogue).

  1. The first takeaway is an obvious one, but it was so beginner-friendly that I had assumed I was above it: when debugging, don't hesitate to call phpinfo();.

  2. The big revelation is an equally simple, though less intuitive, principle: cli !== webserver.

The version of PHP being invoked at terminal (e.g., if I run php --version and so on) is 7.4. By contrast, the version being invoked by the Apache2 web server is 7.3! They do not match! So, as a quick solution, I ended up enabling the PHP 7.4 module and disabling the PHP 7.3 module. I believe I could have stuck with the existing versions, but then I would have had to recompile the mongodb module for PHP 7.3, which could have been a hassle.

Special thanks to da_wunder nim on libera chat for sharing this insight.

  1. The final lesson is not to trust tutorialspoint (lol). Even after I enable the matching version of PHP, I still encounter additional errors in the API for the MongoDB client!

Unfortunately, even this solution caused something else to break (I'm detecting a pattern here). Stay tuned for later when I debug Nextcloud, which broke entirely after the update to PHP 7.4!


Episode III: Docker is great!

I don't know if it's worth telling the full story. Just know that I try running docker and encounter a very difficult bug. After asking the gods on IRC, I am finally able to resolve the error. The problem is very well-explained on another tech blog you'll find here: my-take-on.tech


Episode IV: An SSH Authorization Error (2024-03-30)

I am experiencing a problem whenever I try to "push" commits, either on an existing branch or on a new branch, to a remote git repository on bitbucket. Note that it is a private repository, but this should not matter, since I am using SSH authentication, something I used to do all the time for previous repositories, public and private.

The error message reads:

Pushing to bitbucket.org:workspace/repo.git
Unauthorized
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

The obvious answer is that I have not added my SSH key to the ssh agent. But I have done so, using ssh-add and the infamous eval "$(ssh-agent)" command and updating the known_hosts.

When I try ssh -T git@bitbucket.org, the response is "authenticated via ssh key. You can use git to connect to bitbucket."

I have been trying to follow the suggestions here at cyberciti. I have upgraded git (verison is now 2.30.2).

Even the git remote.origin.url matches the "clone URL" that bitbucket tells me! So this StackOverflow answer does not apply either.

Any advice for me?



Impossible Challenges

OpenPGP

Part I: May 2022

I am exploring a library to use PGP on PHP. However, I run ito a great many difficulties, as I describe below.

So far, I can:

In order to verify the message's authenticity, I use the following example script:

/* Parse public key */
$senderPublicKey = OpenPGP_Message::parse(file_get_contents('ale-public-noascii.key'));

/* Parse signed message from file */
$m = OpenPGP_Message::parse(file_get_contents('signed_message_binary'));

/* Create a verifier for the key */
$verify = new OpenPGP_Crypt_RSA($senderPublicKey);

/* Dump verification information to STDOUT */
var_dump($verify->verify($m));

If I run this basic example, using binary files, then the output is a rather complex JSON dump, but it appears to correctly identify the fingerprint of the public key, as well as the text in the file being verified.

That's a good start, I think to myself. But, for my eventual use-case, I really want to make this process more user-friendly, so I want to allow the user to submit ASCII (base-64-encoded) text as inputs to the function.

So I start to follow the same steps:

But if I simply go to modify the php script to de-armor the files, it fails completely!

I try modifying the example script to use the OpenPGP::unamor method (as in the example below):

/* Parse public key */
$senderPublicKey = OpenPGP_Message::parse(OpenPGP::unarmor(file_get_contents('ale-public.asc'), "PGP PUBLIC KEY BLOCK"));

/* Parse signed message from file named "t" */
$m = OpenPGP_Message::parse(OpenPGP::unarmor(file_get_contents('t'), "PGP SIGNED MESSAGE"));

/* Create a verifier for the key */
$verify = new OpenPGP_Crypt_RSA($senderPublicKey);

/* Dump verification information to STDOUT */
var_dump($verify->verify($m));

But this fails, giving an extraordinarily unclear output:

PHP Notice: Uninitialized string offset: 2050 in /var/www/html/singpolyma-openpgp-php-9651038/lib/openpgp.php on line 471
array(0) {
}

There must be some additional byte-processing calls that I am missing. If you know what these are, please contact me! In the meantime, I have decided to require binary signatures.


Part II: August 2022

As of March 2023, I have finally resolved this issue. However, if you're interested in diving deep into technical nonsense, click to show the problem.

In an effort to write my DRM client, I have been trying to learn and apply the OpenPGP Message specification.

Consider the following example depicting the bytes of a signed message (a properly-generated "gold standard"):

00000000: a301 01da 0125 fe90 0d03 000a 0125 584d .....%.......%XM
00000010: dcd3 4e2b f701 ac13 6206 6568 2e74 7874 ..N+....b.eh.txt
00000020: 627b 73d5 3430 3633 3739 0a89 01b3 0400 b{s.406379......
00000030: 010a 001d 1621 0470 f409 6976 e55e 9d4f .....!.p..iv.^.O
00000040: bda7 b525 584d dcd3 4e2b f705 0262 7b73 ...%XM..N+...b{s
00000050: d500 0a09 1025 584d dcd3 4e2b f776 ff0b .....%XM..N+.v..
00000060: ff72 e481 4e07 b991 0ea8 dcba e6d3 67d8 .r..N.........g.
00000070: 1e5d dbed dd7f 1b8b 1207 459a 55c1 b1dc .]........E.U...
00000080: db8e c7ba 0067 e3c7 69e6 2b60 a781 f6aa .....g..i.+`....
00000090: 8787 a773 c533 d670 645c 9993 6537 c9c4 ...s.3.pd\..e7..
000000a0: f8db 08a6 849e 80c9 fcb2 991f 2ffc 9156 ............/..V
000000b0: abe0 d1a7 c400 8e67 5488 4795 8eaa 028d .......gT.G.....
000000c0: f529 c6be 2c46 5595 9e95 d1f8 2a86 6385 .)..,FU.....*.c.
000000d0: ceee b38b 3e65 8f23 9cdc 168b a8e1 442a ....>e.#......D*
000000e0: 524a 9a0b 3963 b8fd 2d15 b99b cc4b 0a61 RJ..9c..-....K.a
000000f0: b055 18d9 a638 ac5e 76e3 d4f2 ef9b de3f .U...8.^v......?
00000100: 6710 5409 c0c5 44bf 91d7 9e2f c6aa 966b g.T...D..../...k
00000110: 2a64 2233 705b 949a 87ae 883e 5423 8acb *d"3p[.....>T#..
00000120: 7b58 3549 6fcd 2dd1 5c69 bdba 0bdb db99 {X5Io.-.\i......
00000130: 8304 c4ac 2489 7674 920b 7d54 19d1 6f8a ....$.vt..}T..o.
00000140: 157e 138d 4a28 7def c2dd f28e 2209 af8d .~..J(}....."...
00000150: b54f e96d 3829 736a 0a4a 8010 37a5 bfd9 .O.m8)sj.J..7...
00000160: 4197 8856 fe7a c6fd 9e1f b44e 08a8 0eac A..V.z.....N....
00000170: 0853 8977 88c5 4c89 e787 620c 4e8c d77a .S.w..L...b.N..z
00000180: c705 d52a 8579 4b2a f165 5767 8b02 f0d1 ...*.yK*.eWg....
00000190: 0c22 94db 6d3a c1a2 8778 3ce5 6e90 7bd9 ."..m:...x<.n.{.
000001a0: 6739 8941 4bdf 9dd7 352d 90c8 ec53 5015 g9.AK...5-...SP.
000001b0: 130f 01a3 2172 142b 68ac b15a 1456 6c73 ....!r.+h..Z.Vls
000001c0: d28d f99a 5cee 4f63 8307 b493 6582 294b ....\.Oc....e.)K
000001d0: b935 975a 4536 5b4a ab29 cfa3 a492 14b0 .5.ZE6[J.)......
000001e0: 3c <

Converting from Hex to binary, this looks like:

00000000: 10100011000000010000000111011010000000010010010111111110100100000000110100000011000000000000101000000001001001010101100001001101
00000010: 11011100110100110100111000101011111101110000000110101100000100110110001000000110011001010110100000101110011101000111100001110100
00000020: 01100010011110110111001111010101001101000011000000110110001100110011011100111001000010101000100100000001101100110000010000000000
00000030: 00000001000010100000000000011101000101100010000100000100011100001111010000001001011010010111011011100101010111101001110101001111
00000040: 10111101101001111011010100100101010110000100110111011100110100110100111000101011111101110000010100000010011000100111101101110011
00000050: 11010101000000000000101000001001000100000010010101011000010011011101110011010011010011100010101111110111011101101111111100001011
00000060: 11111111011100101110010010000001010011100000011110111001100100010000111010101000110111001011101011100110110100110110011111011000
00000070: 00011110010111011101101111101101110111010111111100011011100010110001001000000111010001011001101001010101110000011011000111011100
00000080: 11011011100011101100011110111010000000000110011111100011110001110110100111100110001010110110000010100111100000011111011010101010
00000090: 10000111100001111010011101110011110001010011001111010110011100000110010001011100100110011001001101100101001101111100100111000100
000000a0: 11111000110110110000100010100110100001001001111010000000110010011111110010110010100110010001111100101111111111001001000101010110
000000b0: 10101011111000001101000110100111110001000000000010001110011001110101010010001000010001111001010110001110101010100000001010001101
000000c0: 11110101001010011100011010111110001011000100011001010101100101011001111010010101110100011111100000101010100001100110001110000101
000000d0: 11001110111011101011001110001011001111100110010110001111001000111001110011011100000101101000101110101000111000010100010000101010
000000e0: 01010010010010101001101000001011001110010110001110111000111111010010110100010101101110011001101111001100010010110000101001100001
000000f0: 10110000010101010001100011011001101001100011100010101100010111100111011011100011110101001111001011101111100110111101111000111111
00000100: 01100111000100000101010000001001110000001100010101000100101111111001000111010111100111100010111111000110101010101001011001101011
00000110: 00101010011001000010001000110011011100000101101110010100100110101000011110101110100010000011111001010100001000111000101011001011
00000120: 01111011010110000011010101001001011011111100110100101101110100010101110001101001101111011011101000001011110110111101101110011001
00000130: 10000011000001001100010010101100001001001000100101110110011101001001001000001011011111010101010000011001110100010110111110001010
00000140: 00010101011111100001001110001101010010100010100001111101111011111100001011011101111100101000111000100010000010011010111110001101
00000150: 10110101010011111110100101101101001110000010100101110011011010100000101001001010100000000001000000110111101001011011111111011001
00000160: 01000001100101111000100001010110111111100111101011000110111111011001111000011111101101000100111000001000101010000000111010101100
00000170: 00001000010100111000100101110111100010001100010101001100100010011110011110000111011000100000110001001110100011001101011101111010
00000180: 11000111000001011101010100101010100001010111100101001011001010101111000101100101010101110110011110001011000000101111000011010001
00000190: 00001100001000101001010011011011011011010011101011000001101000101000011101111000001111001110010101101110100100000111101111011001
000001a0: 01100111001110011000100101000001010010111101111110011101110101110011010100101101100100001100100011101100010100110101000000010101
000001b0: 00010011000011110000000110100011001000010111001000010100001010110110100010101100101100010101101000010100010101100110110001110011
000001c0: 11010010100011011111100110011010010111001110111001001111011000111000001100000111101101001001001101100101100000100010100101001011
000001d0: 10111001001101011001011101011010010001010011011001011011010010101010101100101001110011111010001110100100100100100001010010110000
000001e0: 00111100

Notice that the signed message contains the ASCII plaintext of the signed message as a short segment in the middle!

This gold standard was generated via direct gpg terminal call. Thus, the essence of my investigation (and my bounty) is, what is the best way to replicate this when running my Java program?

This implies that the preceding stream, the first thirty-six bytes of the file, constitute a One-Pass Signature Packet.

Assuming that Chapter 4 of RFC4880 isn't lying and that the message has a packet Header whose first byte includes the "Packet Tag." Well, in that case, We'd have to assume that a3 is our first packet tag. Using the bit interpretation specified in Section 4.2, we can conclude that:

  • the packets are using the "old format;"
  • the tag value is 0b1000, or 8 in decimal;
  • the length value is "of indeterminate length."

If my work so far is correct, then the conclusion of Chapter 4 tells us that this packet is a "Compressed Data Packet" (tag 8) rather than a "One-Pass Signature Packet" (tag 4). Does this make sense? It is possible that the One-Pass Signature (and perhaps other data packets) are wrapped inside a Compressed Packet, but that sure would increase the tedium of this analysis. We'll either have to contact an expert or press forward until we get a clear answer.

I would have expected the tag to be a One-Pass Signature packet; however, it is perhaps noteworthy that "a One-Pass Signature does not interoperate with PGP 2.6.x or earlier." Isn't this a bad sign? Earlier in chapter 4, it reads:

PGP 2.6.x only uses old format packets. Thus, software that
interoperates with those versions of PGP must only use old format
packets. If interoperability is not an issue, the new packet format
is RECOMMENDED. Note that old format packets have four bits of
packet tags, and new format packets have six; some features cannot be
used and still be backward-compatible.

But, look. If we did this correctly so far, then note that Section 5.6 directs us to the "Packet Composition" section to understand how this data is compressed. Section 11.3 is most relevant to us, since this is an OpenPGP message.

As it turns out, this StackExchange answer suggests that my work so far is correct — and that gpg contains a feature to list the packets in a given PGP-related file! And sure enough, when run on our gold-standard file, the first packet is indeed a Compressed Packet! (I didn't bother reading the rest of the output yet, so as to not spoil it.)

To Be Continued...

Takeaways

One upshot of this entire investigation (reading the specification and experimenting) is: were the BouncyCastle experts wrong when they told me to use a LiteralData Packet??? A literal data packet uses tag 11 rather than tag 8, and in Chapter 11.3, Literal Data Packets are shown to comprise Literal Messages rather than Signed Messages and One-Pass Signed Messages. In fairness, in Chapter 5.6 does state that a Compressed Data Packet "contains a literal data packet," but why didn't they explain this missing link to me in their emails?? It doesn't make any sense!



Obscure Linux Distributions



(Un)forgettable Unix Commands

This is a place to store some of the more obscure Linux commands that I often forget.

Server

Desktop

Anywhere (General-purpose)

Finally, take a look at many utilities written by Zaiste in Rust.



FFmpeg Commands

If you've made it this far, congratulations! Here's a GIF I made, using the advice on this forum:



Career Gatekeeping: Year After Year

Or, How They Managed to Lock Me Out of the Good Jobs This Month

Rationale

First, allow me to explain that I have a passion for understanding a system at a deep conceptual level and I also have a fascination with old-fashioned technologies (languages and hardware). These interests lend themselves well to mainframe development. But I have no hands-on experience with these things, because, how would I (how many people have a decades-old mainframe sitting in their basement?). Meanwhile, by pure coincidence, I hear rumors about how people can get paid large sums of money to do work with fortran and COBOL — allegedly, because there are so few programmers who know these languages. Soon after I start recording these narratives in writing, a tech journalist writes that, "if you can wrap your head around this relatively simple but verbose language, you are basically guaranteed a job."

However, as I soon discover, there are no entry-level mainframe positions, as I will attempt to illustrate below.

Ok, well, perhaps this is like the other good jobs in computer science: they demand a few years of experence even for their entry-level positions. Over the course of several years, as I gain more years of experience, the job openings explicitly require an even greater number of years of experience (a moving goalpost). This leads me to the conclusion that the most interesting jobs at the most interesting companies are being reserved for a static set of people: those born before 1997.

So, which is it? Are the jobs paying a high salary because of low Supply, i.e., they cannot find any programmers who know the languages (mere knowledge)? Or are they using that as a pretense, when in reality, they only want highly experienced engineers (several "years of experience")? Or is there an element of ageism, in which Gen Xers "pulled the ladder up behind them?" I have set out to investigate these narratives, by recording data about job vacancies and how their "minimum qualifications" change over time (month after month).

Methodology

The following are actual excerpts from COBOL-related vacancies. I promise you, I did not skew the data by cherry-picking jobs wth unusually high requirements for "prior experience;" rather, these are very representative job openings. If you doubt this, then I challenge you to search it yourself and prove me wrong. In the long run, I intend to integrate a PHP script to automatically append vacancies to this table to drive the point home.

Data

Date Posted Demanded Years of Experience Type of Experience Preferred Position Title Additional Notes
Feb. 20, 2022 4-5 Years "ESB Integration experience" plus
unspecified amnt. of experience with "COBOL, CICS, JCL/PROC, VSAM, DB2, Easytrieve."
Mainframe Developer
Feb. 20, 2022 10 Years (z/OS) "z/OS, TSO, SYSVIEW and/or ISPF, Basic IBM utilities IDCAMS" Mainframe IMS Database Administrator "Candidates that do not meet or exceed the minimum stated requirements (skills/experience) will be displayed to customers but may not be chosen for this opportunity."
The grammar of this sentence is both incorrect andambiguous. I'm surprised they would dare to make it this unclear.
Feb. 20, 2022 5-8 Years* "ESB Integration experience" plus
unspecified amnt. of experience with "COBOL, CICS, JCL/PROC, VSAM, DB2, Easytrieve."
Senior Software Developer,
Technical Lead
*"Experience with COBOL and other programming languages such as [...]"
i.e., the number of years of COBOL is technically unclear.
Feb. 19, 2022 10 Years "The candidate must have the below skills: [...] 10+ years' experience in basic z/OS utilities, TSO, SYSVIEW and/or ISPF, Basic IBM utilities IDCAMS, IEFBR14" Mainframe Administrator
Feb. 19, 2022 7 Years "7+ years of experience in COBOL/JCL/VSAM/CICS." Mainframe Developer
Mar. 22, 2022 7 Years "Must have seven (7) years of experience in the Information Technology field,"
including
"At least five (5) years of experience as a Mainframe SME."
"Proven experience on IBM Mainframe z/OS"
Mainframe Developer
Apr. 19, 2022 10 Years "Hands-on technical experience with mainframe non-x86 legacy systems and with technologies such as COBOL, JCL, CICS, DB2 for z/OS, Assembler, PL/I, Java, Rexx, flat/sequential files, GDGs, and VSAM — 5 years." Mainframe Developer
June 8, 2022 7-12 Years "Minimum of 3 years of experience working on mainframe, SQL and Relational Databases such as DB2, Oracle, and SQL Server." Systems Analyst The mainframe-specific year threshold is more attainable, for once, but they still insist on seven years of professional experience.
August 26, 2022 20 Years "opportunity to work on the same systems you used to as a kid: IBM Mainframe; IDMs; COBOL; [...]" Software Engineer
September 23, 2022 Bachelor's Degree + 3 Years "Autosys: 3 years (Required)"
"z/OS Mainframe: 3 years (Required)."
"Cobol: 3 years (Required)"
Infrastructure Governance Deployment Technician

Notice that many positions give ambiguous descriptions that do not give an exact number of years for COBOL specifically, but nonetheless state that it is required. Examples include: "Experience with mainframe and Cobol is a plus" (Edward Jones); "Experience with COBOL and other programming languages such as [...]" (Zigabyte); "Experience with COBOL, CI/CD, DB2 is required" (Benedsoft).



© 2024

Font credits

MIT License
Copyright (c) 2019 AKIRA-MIYAKE
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Copyright (c) 2014, The Fira Code Project Authors (https://github.com/tonsky/FiraCode)
This Font Software is licensed under the SIL Open Font License, Version 1.1. This license is copied below, and is also available with a FAQ at: http://scripts.sil.org/OFL

SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007

PREAMBLE The goals of the Open Font License (OFL) are to stimulate worldwide development of collaborative font projects, to support the font creation efforts of academic and linguistic communities, and to provide a free and open framework in which fonts may be shared and improved in partnership with others.

The OFL allows the licensed fonts to be used, studied, modified and redistributed freely as long as they are not sold by themselves. The fonts, including any derivative works, can be bundled, embedded, redistributed and/or sold with any software provided that any reserved names are not used by derivative works. The fonts and derivatives, however, cannot be released under any other type of license. The requirement for fonts to remain under this license does not apply to any document created using the fonts or their derivatives.

DEFINITIONS "Font Software" refers to the set of files released by the Copyright Holder(s) under this license and clearly marked as such. This may include source files, build scripts and documentation.

"Reserved Font Name" refers to any names specified as such after the copyright statement(s).

"Original Version" refers to the collection of Font Software components as distributed by the Copyright Holder(s).

"Modified Version" refers to any derivative made by adding to, deleting, or substituting -- in part or in whole -- any of the components of the Original Version, by changing formats or by porting the Font Software to a new environment.

"Author" refers to any designer, engineer, programmer, technical writer or other person who contributed to the Font Software.

PERMISSION & CONDITIONS Permission is hereby granted, free of charge, to any person obtaining a copy of the Font Software, to use, study, copy, merge, embed, modify, redistribute, and sell modified and unmodified copies of the Font Software, subject to the following conditions:

1) Neither the Font Software nor any of its individual components, in Original or Modified Versions, may be sold by itself.

2) Original or Modified Versions of the Font Software may be bundled, redistributed and/or sold with any software, provided that each copy contains the above copyright notice and this license. These can be included either as stand-alone text files, human-readable headers or in the appropriate machine-readable metadata fields within text or binary files as long as those fields can be easily viewed by the user.

3) No Modified Version of the Font Software may use the Reserved Font Name(s) unless explicit written permission is granted by the corresponding Copyright Holder. This restriction only applies to the primary font name as presented to the users.

4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font Software shall not be used to promote, endorse or advertise any Modified Version, except to acknowledge the contribution(s) of the Copyright Holder(s) and the Author(s) or with their explicit written permission.

5) The Font Software, modified or unmodified, in part or in whole, must be distributed entirely under this license, and must not be distributed under any other license. The requirement for fonts to remain under this license does not apply to any document created using the Font Software.

TERMINATION This license becomes null and void if any of the above conditions are not met.

DISCLAIMER THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.