[jacking off motion] great 🙄
-
"extensive custom typemaps"
I didn't realize the models would glaze up source code it's told to analyze, too
(there are very few custom typemaps and they are anything but "extensive")
-
the set of models available to us has been "upgraded" so now we're able to access the latest-and-greatest
trying the same tasks as yesterday. the one that is "describe [thing I know well] to me" has been spinning for quite a while now. I knew that getting responses from models had become very slow, but I did not realize it was "20 seconds to `cat` 1k lines into a file" slow
how do you even accomplish that
@SnoopJ The Aristocrats!
-
@arrjay I didn't look too hard at the JS source embedded in the CLI but I could easily believe React is in there based on the quality of GitHub's work in general
@arrjay okay now I looked and I don't see React in there but I guess I can't *really* rule it out
-
(there are very few custom typemaps and they are anything but "extensive")
```
Python Dependencies:
- Optional: pathlib (stdlib)
```uh
-
```
Python Dependencies:
- Optional: pathlib (stdlib)
```uh
@SnoopJ That is... confusing on *multiple* levels.
-
@SnoopJ That is... confusing on *multiple* levels.
@xgranade I'm pretty sure I know how it fucked this one up: the bindings in question have a wrapper for the internal `Acme::Path` and most methods that accept paths will take an `AcmePath` or a `PyPath` (disambiguation of `pathlib.Path`)
It's *wrong*, but I think I can see why
-
@xgranade I'm pretty sure I know how it fucked this one up: the bindings in question have a wrapper for the internal `Acme::Path` and most methods that accept paths will take an `AcmePath` or a `PyPath` (disambiguation of `pathlib.Path`)
It's *wrong*, but I think I can see why
@SnoopJ I mean, that gets at the "optional" part, but why is it listing dependencies on the *standard library* at all?
-
@SnoopJ I mean, that gets at the "optional" part, but why is it listing dependencies on the *standard library* at all?
@xgranade yea, indeed
-
```
Python Dependencies:
- Optional: pathlib (stdlib)
```uh
Verdict on the code analysis task is that it didn't fuck up its explanation anywhere near as much, although it still fucked up plenty enough to be dangerous to someone who is asking because they don't already know the answers.
Turned the model loose on the "merge these two documents" task and it finished working in a handful of minutes. Now I get to go over the generated result with a fine-tooth comb and see if it's any good.
-
Verdict on the code analysis task is that it didn't fuck up its explanation anywhere near as much, although it still fucked up plenty enough to be dangerous to someone who is asking because they don't already know the answers.
Turned the model loose on the "merge these two documents" task and it finished working in a handful of minutes. Now I get to go over the generated result with a fine-tooth comb and see if it's any good.
@SnoopJ see if it's any good*
*: see if it's any good _this time_
-
@SnoopJ see if it's any good*
*: see if it's any good _this time_
@glyph yes, I mean this specific output
-
Verdict on the code analysis task is that it didn't fuck up its explanation anywhere near as much, although it still fucked up plenty enough to be dangerous to someone who is asking because they don't already know the answers.
Turned the model loose on the "merge these two documents" task and it finished working in a handful of minutes. Now I get to go over the generated result with a fine-tooth comb and see if it's any good.
Well, I've already spotted mistakes, so let's have clod chew on it some more with directions to unfuck it
I really resent the way this CLI handles permissions. For editing a single file, you have a choice between "allow this specific edit operation this one time" and "allow all file operations for the rest of this session on any files"
A "you can touch *that* file as much as you like" option seems like an obvious thing to add if you give a shit about limiting the blast radius of letting the model use external tools, but I guess that's a bridge too far huh
-
Well, I've already spotted mistakes, so let's have clod chew on it some more with directions to unfuck it
I really resent the way this CLI handles permissions. For editing a single file, you have a choice between "allow this specific edit operation this one time" and "allow all file operations for the rest of this session on any files"
A "you can touch *that* file as much as you like" option seems like an obvious thing to add if you give a shit about limiting the blast radius of letting the model use external tools, but I guess that's a bridge too far huh
no wonder Enthusiasts end up nuking their shit, I wouldn't want to babysit the thing accepting each atomic operation as they come either, with how slow this process is
and the only other alternative is "fuck my shit up as much as you want"
-
Well, I've already spotted mistakes, so let's have clod chew on it some more with directions to unfuck it
I really resent the way this CLI handles permissions. For editing a single file, you have a choice between "allow this specific edit operation this one time" and "allow all file operations for the rest of this session on any files"
A "you can touch *that* file as much as you like" option seems like an obvious thing to add if you give a shit about limiting the blast radius of letting the model use external tools, but I guess that's a bridge too far huh
@SnoopJ IT, understanding consent, we've been here
-
@SnoopJ IT, understanding consent, we've been here
@arrjay sucks here
-
no wonder Enthusiasts end up nuking their shit, I wouldn't want to babysit the thing accepting each atomic operation as they come either, with how slow this process is
and the only other alternative is "fuck my shit up as much as you want"
Verdict: a more powerful model is capable of doing the dumb parts of the document concatenation, but nursing it through making fine edits for that concatenation to make sense is worst than just editing the document by hand.
So, I don't think it's very good at this kind of paperwork, either. Maybe for a document nobody cares about.
-
@glyph yes, I mean this specific output
-
-
Verdict: a more powerful model is capable of doing the dumb parts of the document concatenation, but nursing it through making fine edits for that concatenation to make sense is worst than just editing the document by hand.
So, I don't think it's very good at this kind of paperwork, either. Maybe for a document nobody cares about.
and doing so took the model ~336,500 tokens
For reference, the final merged document is about 20 KB of text, so conservatively about 8 tokens per byte processed (assuming I started with 2x 20 KB docs which is overestimating)
Woof.
-
and doing so took the model ~336,500 tokens
For reference, the final merged document is about 20 KB of text, so conservatively about 8 tokens per byte processed (assuming I started with 2x 20 KB docs which is overestimating)
Woof.
@SnoopJ@hachyderm.io what's the cost of a token?
