X-Git-Url: https://scripts.mit.edu/gitweb/www/ikiwiki.git/blobdiff_plain/68fd245addb9b96b473949349440b86796955cce..45c31be7dbf81517e854304237bb95d53657b27d:/doc/todo/should_optimise_pagespecs.mdwn diff --git a/doc/todo/should_optimise_pagespecs.mdwn b/doc/todo/should_optimise_pagespecs.mdwn index 3dfa8e1f2..728ab8994 100644 --- a/doc/todo/should_optimise_pagespecs.mdwn +++ b/doc/todo/should_optimise_pagespecs.mdwn @@ -79,8 +79,6 @@ I can think about reducung the size of my wiki source and making it available on > > --[[Joey]] -[[!template id=gitbranch branch=smcv/ready/optimize-depends author="[[smcv]]"]] - >> I've been looking at optimizing ikiwiki for a site using >> [[plugins/contrib/album]] (which produces a lot of pages) and it seems >> that checking which pages depend on which pages does take a significant @@ -90,6 +88,8 @@ I can think about reducung the size of my wiki source and making it available on >> rather than a single pagespec. This does turn out to be faster, although >> not as much as I'd like. --[[smcv]] +>>> [[Merged|done]] --[[smcv]] + >>> I just wanted to note that there is a whole long discussion of dependencies and pagespecs on the [[todo/tracking_bugs_with_dependencies]] page. -- [[Will]] >>>> Yeah, I had a look at that (as the only other mention of `pagespec_merge`). @@ -98,8 +98,6 @@ I can think about reducung the size of my wiki source and making it available on >>>> I haven't actually deleted it), because the "or" operation is now done in >>>> the Perl code, rather than by merging pagespecs and translating. --[[smcv]] -[[!template id=gitbranch branch=smcv/ready/remove-pagespec-merge author="[[smcv]]"]] - >>>>> I've now added a patch to the end of that branch that deletes >>>>> `pagespec_merge` almost entirely (we do need to keep a copy around, in >>>>> ikiwiki-transition, but that copy doesn't have to be optimal or support @@ -235,6 +233,23 @@ master at the time of rebasing: 14.20s to rebuild, 10.04/12.07/14.01s to refresh. I think you can see the bug clearly here - the pagespecs are getting more complicated every time! +> I can totally see a bug here, and it's one I didn't think existed. Ie, +> I thought that after the first refresh, the pagespec should stabalize, +> and what it stabalized to was probably unnecessarily long, but not +> growing w/o bounds! +> +> a) Explains why ikiwiki.info has been so slow lately. Well that and some +> other things that overloaded the system. +> b) Suggests to me we will probably want to force a rebuild on upgrade +> when fixing this (via the mechanism in the postinst). +> +> I've investigated why the pagespecs keep growing: When page A changes, +> its old depends are cleared. Then +> page B that inlines A gets rebuilt, and its old depends are also cleared. +> But page B also inlines page C; which means C gets re-rendered. And this +> happens w/o its old depends being cleared, so C's depends are doubled. +> --[[Joey]] + After the initial optimization: 14.27s to rebuild, 8.26/8.33/8.26 to refresh. Success! @@ -243,17 +258,56 @@ I'm worried that duplicates will just build up (again) in less simple cases, though, so 0.2s is probably a small price to pay for that not happening (it might well be experimental error, for that matter). +> It's weird that the suggested optimisations to +> `add_depends` had no effect. So, the commit message to +> b6fcb1cb0ef27e5a63184440675d465fad652acf is actually wrong.. ? --[[Joey]] + +>> I'll try benchmarking again on the non-public wiki where I had the 4% +>> speedup. The docwiki is so small that 4% is hard to measure... --[[smcv]] + Not saving {depends} to the index, using a hash instead of a list to de-duplicate, and allowing add_depends to take an arrayref instead of a single pagespec had no noticable positive or negative effect on this test. +> I see e4cd168ebedd95585290c97ff42234344bfed46c is still in your branch +> though. I don't like using an arrayref, it could just take `($page, @depends)`. +> and I don't see the need to keep it if it doesn't currently help. + +>> I'll drop it. --[[smcv]] + +> Is there any reason to keep 7227c2debfeef94b35f7d81f42900aa01820caa3 +> if it doesn't improve speed? +> --[[Joey]] + +>> I'll try benchmarking on a more complex wiki and see whether it has a +>> positive or negative effect. It does avoid being O(n**2) in number of +>> dependencies. --[[smcv]] + Memoizing the results of pagename brought the rebuild time down to 14.06s and the refresh time down to 7.96/7.92/7.92, a significant win. +> Ok, that seems safe to memoize. (It's a real function and it isn't +> called with a great many inputs.) Why did you chose to memoize it +> explicitly rather than adding it to the memoize list at the top? + +>> It does depend on global variables, so using Memoize seemed like asking for +>> trouble. I suppose what I did is equivalent to Memoize though... --[[smcv]] + Refactoring to use pagespec_match_list looks more risky from a code churn point of view; rebuild now takes 14.35s, but refresh is only 7.30/7.29/7.28, another significant win. --[[smcv]] +> I had mostly convinced myself that +> `pagespec_match_list` would not lead to a speed gain here. My reasoning +> was that you want to stop after finding one match, while `pagespec_match_list` +> checks all pages for matches. So what we're seeing is that +> on a rebuild, `@changed` is all pages, and not short-circuiting leads +> to unnecessary work. OTOH, on refresh, `@changed` is small and I suppose +> `pagespec_match_list`'s other slight efficiencies win out somehow. +> +> Welcome to the "I made ikiwiki twice as fast +> and all I got was this lousy git sha1sum" club BTW :-) --[[Joey]] + [[!tag wishlist patch patch/core]]