Making progress. Trying to do an hour or two of test writing a day. Have three SMR tests passing now.

Once the SMR tests are complete, need to apply SMR to the test suite so that CAS can be tested.

Something – I think a robot of some kind – keeps buglist.cgi running constantly, which is massively slowing down the site.

For this moment, I’ve renamed buglist.cgi. I need to look into the problem in more detail to discover the real origin.

So, now need to write a test suite for SMR, then use SMR API in the test programme so that CAS versions of the data structures can be tested.

After a day of thinking it over, I’ve opted to ditch all allocation and have the user provide NUMA-allocated store for SMR.

Test Iteration 01
=================

Abstraction Tests
=================
Atomic increment…passed
Atomic DCAS…passed
Atomic CAS…passed

Add-Only Unbalanced BTree Tests
===============================
Random adds and walking…passed

Add-Only Hash Tests
===================
Random adds…passed
Iterate…passed

Add-Only SList Tests
====================
New head…passed
New tail…passed
New after…passed
New ordered…passed
New ordered with cursor (10 seconds)…passed

Freelist Tests (DCAS)
=====================
Popping…passed
Pushing…passed
Popping and pushing (10 seconds)…passed
Rapid popping and pushing (10 seconds)…passed

Queue Tests (DCAS)
==================
Enqueuing…passed
Dequeuing…passed
Enqueuing and dequeuing (10 seconds)…passed
Rapid enqueuing and dequeuing (10 seconds)…passed

Ringbuffer Tests (DCAS)
=======================
Reading…passed
Writing (10 seconds)…passed
Reading and writing (10 seconds)…passed

Stack Tests (DCAS)
==================
Popping…passed
Pushing…passed
Popping and pushing (10 seconds)…passed
Rapid popping and pushing (10 seconds)…passed

Now to sort out SMR.

So, spambots figured out the answer the human-only questions and made about 60 accounts. They’ve only made their own talk pages, so it’s not too painful, but now I want to delete those accounts.

This cannot be done.

Mediawiki is surprisingly shockingly bad at any kind of mass operation or deleting.

There’s support out of the box for single user blocking, but who wants to manually block sixty users, one after the other? and what when there are many hundreds?

After searching, I found in the end three extensions which claim mass user delete or at least block. Not a one of them worked. (There’s a few more in fact, but they all have large warnings, like cigarette cartons – THIS EXTENSION WILL DAMAGE YOUR DATABASE).

Mediawiki is unable to handle spam attacks once they have occurred. There are some good extensions to help prevent attacks, but – once they’ve happened, and you’ve got the mess, you’re pretty much stuffed.

I don’t understand this. The whole point of mediawiki is for multiple users and fairly open access. Spam attacks *are* going to occur. There *has* to be support for cleaning up the mess. What are the mediawiki authors doing? I question their competence.

Well, I’ve made a hack to have ringbuffer work.

Each ringbuffer element contains both an actual allocated queue element and a pointer to a queue element. The pointer is set when the ringbuffer element is dequeued, e.g. it doesn’t use its own actual queue element, but rather the element dequeued from the queue.

This breaks memory locality, because over time the queue element being used will be some random other element, rather than in the same element.

What’s needed is a queue with no dummy element. These exist, but I do not want to suspend the release of 7.0.0 to implement such a thing!

So this hack will allow release, while also allowing a ringbuffer with user controlled memory allocation. A future release will implement a non-dummy element queue and there memory locality for ringbuffer will be attained.

I joined a startup at the end of Nov 2012.

Been kinda focused on that since then :-)

Just finished the main internal data system.

Thinking of looking again a bit at liblfds. Sadly, I think there’s a good couple of man weeks of work left to do before release.

Will have a bit of a look today and see if I can remember where things had come to.

I’ve been working on making the tests pass.

Everything went fine until I came to the penultimate test, ringbuffer writes.

This fails. It’s a peculiar failure; when we come to get a write element, we find both the freelist and the queue are empty. This is in this test impossible – there are 100,000 elements per thread, each thread can at most have one element taken out of the ringbuffer, the balance of the elements will either be in the freelist or the queue. So what gives?

Turns out I had forgotten how the queue works.

The queue has a dummy element. It’s not a static dummy element, though; what it is, is that the element pointed to by the dequeue pointer has been emptied already of its user data – when you perform a dequeue, you get the user data from dequeue->next, not from dequeue itself.

E.g.

dequeue
|
element1 – element2

element1 is the dummy – his user data has already been returned to the user; when we dequeue, we’ll get the user data from element2.

Now, the problem is that for the ringbuffer, I have in each ringbuffer element a queue_element; and when I init the queue, I pass in an extra element, to be the initial dummy.

So I assume when I dequeued, the queue element I obtained in the ringbuffer would be the queue element used for that ringbuffer element. Problem is, it isn’t! it’s the queue element *prior* in the queue!

Benchmark compiles (again – I did some more work on it).

I’ve now been going through the tests, making them work.

Freelist, queue and stack all pass for DCAS.

Ringbuffer blows a gasket initalizing its freelist. That’s next on the list.

Once DCAS is up I may then get benchmark actually working; either that or proceed with sorting out SMR.