Paintball research respository

[FatMan]

After reading and posting to Deep Blue for a while, it seems to be it would be really nice to have a repository of paintball research results. Maybe this could be a classic thread with restricted postings or maybe an area in AO separate from the forums, or maybe on another site altogether.

What I'd like to see is not ramblings or pontificating, but experimental data. Results from experiments, simulations, etc. I'm not sure who all would be willing to share their results, but I know we have HEARD about a lot of tests. It would be really nice if these were listed in one place where we could find them and read and reread them as we wanted to.

For example:
- there's the bounce test data that AGD and others have produced.
- there's some accuracy tests posted on <another site> that we could post a link to
- there are all sorts of mythical tests AGD has run and talked about, but maybe we could get them to record them in a little more detail

From time to time us Deep Bluers are inspired to do more of this. Maybe such a repository would prompt more people to do that and post it.

What do you guys think?

FatMan

[bjjb99]

I like the idea. In addition to results, one could post testing procedures that were used so other folks could repeat the experiments (or modify them for different markers, paint, etc.). I also see a section on the actual design of experiments... folks collaborating to develop rigorous testing procedures following scientific methodologies.

Now that I've used a bunch of big words, I'm gonna take a nap. ;)

BJJB

[AGD]

Not a bad idea. We should post the data in two places. One in the thread it's being discussed and a second in an independent thread that I can lock. As time goes on I can merge all the data threads into one.

AGD

[FatMan]

Originally posted by AGD
Not a bad idea. We should post the data in two places. One in the thread it's being discussed and a second in an independent thread that I can lock. As time goes on I can merge all the data threads into one.

AGD

Yeah, that's what I had in mind. Folks can post their data in Deep Blue as a regular thread, and AGD can move selected items to a locked thread for posterity - only the data any any peritent updates, not all the discussion.

Also, as bjjb99 pointed out, good postings should have their testing procedures spelled out as well as both raw and cooked data.

FatMan

[FatMan]

As an example, I've been reading the thread about barrel bore size. I'm thinking about doing the following experiment:

Mount my Mag in a bench mount (cphillip has one that should do). Select three different paints: NelSplat (kindof big), Powerball (kindof medium), and Proball Platinum (kindof small). For each paint I do the following:

1) bounce test.
2) see which insert from my Freak set seems to be the best. - have cphillip do the same (independently)
3) starting with the smallest Freak insert, fire 10 rounds at a clean target. Measure velocity of each shot. Afterwards, measure the center of the group and record the distance of each shot from the center.
4) repeat with all 8 inserts.
5) if I'm not bored silly by then, repeat the whole thing with the different length barrel tip (between Phil and I we have a 10" and 12").

Two tips times 8 inserts times 3 paints times 10 rounds is 480 data points.

Probably the biggest problem will be breaks. Of course I want to record breaks, but then I need to get the barrel back to clean before I continue. If I'm willing to assume two Freaks backs are the same (probably OK since the inserts control the ID) I should be able to remove a dirty one and clean it while using the other. I also want to be sure to fire a few rounds through a recently cleaned barrel to "age" the cleanliness.

Then we get to cook the data and see what we get. I'm thinking average velocity, standard deviation of velocity, group size, average distance from center, standard deviation of distance from center, number of breaks.

Boy, will I be disappointed if it all comes back that it makes no difference at all! I mean, after gathering all that data I'd like to see SOMETHING! :rolleyes:

What do you think? Am I missing something? Anyone out there want to repeat the experiment on different markers (Angle, Cocker, etc) or with a different barrel system (like a set of Boomies)?

Once thing I'm unclear about. I plan to leave the pressure set the same for all tests - chrono'd close to 300fps. One thing I'd like to see is how the barrel afects velocity - and thus efficiency. But is that fair in assessing accuracy? Should I do two sets of tests - one for velocity where I leave the pressure set and one for accuracy where I re-chrono to 300fps for each test? Makes sense, but its a lot more work!

Comments?

FatMan

[than205]

I would think you would be more interested in the data aquired from the maintained fps than preset fps. While any data acquired regarding bore size changes are interesting, if you don't adjust (correct) the velocity you won't be getting any real world data. Meaning, the average player whether they have five barrels or one barrel will adjust velocity as needed. Besides what barrel would you use for your benchmark? (crown point? hehehe)

My concern for the repository would setting guidelines for the non-scientific to contribute. Not everyone has a grasp of the scientific process.

[AGD]

Fot this to make sense you would have to publish your protocall first and have it approved by the group. For instance firing 10 shots from each size is not statistically significant, you need about 50 or more. You should also process your data in statistical format and give a confidence value in how different two groups of velocities are .

AGD

[Redkey]

It would probably be a good idea to have some type of a peer review process for the results before they are published... or better yet, before the tests are run.

I'd hate to be the guy who shoots 500 shots recording every detail about them only to find out he missed some important issue that makes his results meaningless.

How many people would actually be able to perform testing like this in a controlled and repeatable manner? From what I've read AGD has a test setup and I guess warpig has something going as well.

As for Fatmans tests...
I would also include the weight of the individual paintballs

You should probably establish a standardized method of cleaning the barrel.

photos of the test setup would be a good idea too. Actually, if you take photos of the paint marks on the target, image analysis can be used to measure the relative locations of the shots to each other and the center of the target.

How will you control the aimpoint of the marker? If you're going to include the average distance from center you'll need to make sure the gun is aimed the exact same location every time.

bjjb99... are you talking about designing experimental procedures or design of experiments (DOE) which is a statistical method of looking for interactions between variables without actually running all the tests to examine every possible combination of inputs.

Anyhow... I'm interested in this and would be willing to help people with their test procedures and statistical analysis work.

[bjjb99]

FatMan wrote:
>
> What do you think? Am I missing
> something? Anyone out there want to
> repeat the experiment on different
> markers (Angle, Cocker, etc) or with a
> different barrel system (like a set of
> Boomies)?

I agree with Tom regarding the number of shots fired. You really want much more than ten per insert. We use 32 elements as the bare minimum for statistical sample sets; usually the number is a hundred or more.

I assumed you were planning on adjusting the marker's velocity so that each insert chronographed the same, at least for the accuracy tests. For consistancy testing, you may not want to adjust the velocity at all... just set the mean velocity at 300 fps (or 290, 280, whatever) for the insert that best fits the paint and see how that velocity changes when different diameter inserts are used.

You will also want to measure the meteorlogical conditions during the test. A few data points (before, twice during, and at the end) should be enough for temperature, relative humidity, and barometric pressure. More would be needed for wind speed and direction. Of course, the ideal situation would be testing indoors.

For each insert, you could do the talcum powder visualization test that I mentioned in the barrel diameter thread; lightly coat the inside of the Freak insert with powder and fire a paintball through it... then examine the streaks left in the powder where the ball contacted the barrel. You can photograph a few for each insert diameter to determine if the ball behaves differently as a function of ball-barrel match.


Redkey wrote:
>
> bjjb99... are you talking about
> designing experimental procedures or
> design of experiments (DOE) which is a
> statistical method of looking for
> interactions between variables without
> actually running all the tests to
> examine every possible combination of
> inputs.

I was thinking more along the lines of the actual designing of experimental procedures. I haven't done much, if any, DoE work, though I can see where it would really come in handy when the number of experimental variables becomes large.

BJJB

[FatMan]

Great comments all! :cool:

I think we should try to give feedback to potential experimenters - just like you guys just gave to me. There would be extra incentive to repeat experiments with the same setup. At the same time, this isn't going to be the most scientific testing in the world. I mean, for the most part we're a bunch of paintball guys who are trying to put some reality into some of the myths of paintball. We don't all have perfect lab facillities or a budget for 5000 rounds of test paint.

I guess what I'm saying is probably any data can be criticized to some extent and we should not be TOO hard nosed about it. To the extent we feel some data is useful we should include it - while pushing for better testing procedures. It seems that right now we have nothing. If we have a bunch of data, then there's no point in adding more unless the data is somehow better (by virtue of the procedure or statistical method).

I think we should encourage folks to propose their procedure first, and then present their data, and then based on response decide what should be archived. Rather than make a bunch or rules, let's see what people come up with.

Anyway, as for specific comments:

I agree on the number of shots thing. I hadn't thought about numbers so much yet.

I like the standard cleaning procedure (I was hinting around at that) and weighing the balls seems a reasonable thing. I was thinking of photos also.

As for the aimpoint - I was planning to compute the center of the group for each test and then measure deviation from the center - I'm more interested in consistency than absolute aim. That was also why I wasn't so concerned with velocity, but if velocity varies too much it could have an unfair effect on apparent consistency.

Environmental conditions I was planning to record also. I don't have an indoor range, but I might be able to put in some wind breaks to reduce the effect of wind gusts.

As for the powder stuff, I think that's cool, but it would be a real pain to do THAT several hundred times - so I'm thinking I'll leave that for someone else to do. I guess if everyone things we MUST do that we could consider it ...

Wow, this is really cool. Now I'm really fired up to do some of this. Not sure how quickly I can get everything together to get started, but as soon as I can ...

Now let's see who ELSE wants to get some experiments going. Bjjb, weren't you going to be doing some stuff? :D

FatMan

[FatMan]

So, I'd like to propose that a copy of AGD's pressure curve graph at:

http://www.automags.org/forums/showthread.php?s=&threadid=18939&pagenumber=2

Be added to the research repository. I'd also like to consider the posting AGD made at:

http://www.automags.org/forums/showthread.php?s=&threadid=34109

While its not lab data, it IS a report of a well controlled study relative to the discussions here.

Any opinions on the matter?

FatMan

[FatMan]

First, thanks AGD for posting the Official Data thread!

Next, I've been getting prepared to run the experiments I've proposed. I think I have everything covered except the ball weight issue. I'm having trouble finding a scale that will weigh something that light with reasonable precision. I'm sure they exist around here, but getting someone to let me borrow one is another issue.

How valuable would the proposed data be without that? Obviously we would not be able to correct for variations in weight. Does anyone have actual data that shows that the weight variation is significant? What do you all think?

As for recording the shot data, I'm thinking of using a large sheet of paper with a 1 inch grid and recording the coordinates of each shot. From there I can calculate the centroid of the group, and then the distance from the centroid for each shot. Any thoughts?

50 shots to one sheet of paper would probably result in a mess. I was thinking of doing groups of 10 or 15 or so. I can probably get each sheet of paper mounted pretty close to the same way each time - or I can only calculate the centroids for each subgroup. What do you think? Both approaches will have some error, but which would you think is worse?

Finally, on velocity. Everyone seems to agree I need to adjust the velocity each time I change the barrel. So the question is - what procedure for doing that. As we all know, shot-to-shot velocity varies - so how do I determine that I have returned it to the "same" velocity each time? This might be further compounded by the fact that shot-to-shot variation may be different with different barrel inserts. Ideas would be appreciated.

I'm hoping to get started this weekend. I will probably only get to do one paint this weekend, but that should be a good start. I have pretty much everything, except the scale, ready to go.

One last issue. The only chrono I have is an optical. I know those aren't as accurate as a radar model, but I just don't have access to one of those. Will this compromise the experiment? Thoughts?

FatMan

[bjjb99]

Rather than using paper for your target (it will turn into a soggy mess and will likely be perforated by the paintball impacts), you might want to consider some sort of plastic sheet. If you don't mind plunking down some dollars, you can get a 4x4 foot piece of 1/8" thick transparent polycarbonate for around $50 plus shipping. Paint or otherwise mark lines on the backside of the sheet so that they form 1" squares, hang it against a white-painted plywood back, and you've got a target that you can wash/wipe off after each test.

If the price is a bit steep, you can go with polyethylene sheet. It's a bit less rigid and it's usually not nearly as transparent, but it costs about half as much as polycarbonate for the same size piece. You could also probably get away with a 2x2 foot section instead of a 4x4, depending on the target distance you want to use for testing.

Plastic sheet is sold in various places online. McMaster Carr is always a good place to start looking. Just put "plastic sheet" into their search engine and away you go. Here's their URL:

http://www.mcmaster.com/

I would suggest adjusting the velocity for each insert until the mean of several shots matches the desired value, give or take one standard deviation? Then perform the centroid test and measurement?

BJJB

[bjjb99]

Probably your best bet (aside from spending more dollars than you want to) would be a college chemistry department. See if you can borrow one of their triple beam balances, which generally have 0.1 gram sensitivity (that's about 3 percent of an average paintball's mass). If you want more precision than that, you'll probably have to go to an electronic toploading balance. You may have to perform your paintball weighings in a chem lab and then take the paint to your firing range. Just make sure the paint doesn't soak up a lot of humidity during transport, and you should be ok.

Regarding the chronograph... I think for a first run of the experiment an optical chronograph is a reasonable choice. After working out some of the kinks in the testing it may become necessary to go to something more precise, but not yet.

Be sure to note the make and model of your test equipment (what type of scale for weighing paint, the type of chronograph, etc.) so other folks can repeat the test using hardware built to identical specs if they wish to do so.

BJJB

[FatMan]

Hmm, if a triple beam will do the trick, I can probably swing that. i was worried I'd need something more sensitive. Now who do I know in chemistry ...?

I like the polycarb sheet idea. A little pricey. Maybe I can work up something that will do just as good though. Let me think about that. There is likely to be something like that lying around I can scrounge.

Thanks for the ideas,

FatMan

[cledford]

FatMan,

Try a precison digital powder scale. They're used for weighing powder charges when reloading firearm ammunition. They can measure in either grains (fractions of grams, 1 grain = .0647 grams) or grams.

Here are the parameters for the pact version:

1500 Grain or 100 Gram Capacity.
.1 Grain or .01 Gram Resolution.
+- .1 Grain .01 Gram Accuracy.

I've got one and it works great.

You can find more info at Pacts website:

http://www.pact.com/disp.html

-Calvin

[AGD]

Since we have years of experience doing this I'll give you some hints. First of all the weight difference between paintballs is nothing to worry about and you would need a scale that does 10ths of a gram anyway.

Use large sheets of paper for the targets with no backing. Tape them into a frame with each corner marked so each paper can be aligned and taped in the same postition. Fire 10 rounds and then go down to the paper and put an x across each tear hole and make sure you come up with 10 x's. QUite often balls go through the same tear. For precision groups we have someone stand down range behind plywood and they come out and mark each impact point since they can see where it hit. Enter the x,y coordinates in Excel to get your statistics. Do not bother creating a 1 inch grid for each paper use a yardstick to measure each x.

AGD

[FatMan]

Gee, I hope you don't get tired of hearing folks say "thanks!" ;)

Well, I had just managed to find a place to borrow a scale with .001 gram precision (thanks for the idea bjjb - I had one of those once). On the other hand, if ball weight variation is just going to be noise it would sure speed up the test procedure to skip that. I'm willing to accept AGD's experience on this unless everyone thinks I should do it to be SURE I can account for ball weight variation.

As for the paper idea, that sounds like a good idea. I wasn't going to create the grid, but try to get paper already gridded. If I can't get that, then direct measure would work fine.

Now, my PLAN had been to record weight, speed, and offset for each ball (keeping the three stats together per ball). That would seem to require identifying the hole ball-for-ball. I *could* just get the stats on the speeds and offsets (and weights if I did that) and report averages, std devs, etc. I'm not sure that the detail data would really tell us anything useful.

Once again, I ask you - do I need to identify which velocity went with which offset (and weight) - which would take more effort - or just gather all the data for aggregation. Sounds like AGD is recommending the latter.

As a compromise, I could use the less detailed procedure until I identify the BEST barrel insert (assuming there is one) and THEN do a detailed study of THAT to see if things like ball weight account for whatever variation remains.

Its a thought! Anyway, I'mm still gearing up to start this - so any input from you guys is welcome. And AGD, thanks again for your input!

FatMan

[Redkey]

When I was working up loads for my 357 I was performing accuracy tests for bullet and powder load combinations. Since the measuring the shot spread was a hassle I setup an image analysis package to make the measuremnts for me.

A digital image of the impact points is fed into the software which finds the holes due to their contrast with the white paper (I put something black behind the holes). Then the software grinds away to determine the center of mass for each impact and spits out x, y coordinates based on what ever measurement system it was calibrated for. It would also kick out the coordinates of the bullseye for a reference point.

In excel I calculated the distance from the bull for each shot and then used the average and standard deviation of the spread to make the load to load comparisons.

There are several free IA packages out there, if anyone is interested I could look around and find something that will work for this type of measurement. Mine has a dongle so I cannot share.

In the short term the ruler method will always be quicker and just as accurate... the IA method is neat because the measurements can also be taken real-time with a video camera after each shot allowing you to link the impact point with what ever other data has been collected by the computer... velocity, pressure profiles, wind speed etc.

Call me a geek but half the fun is building the test system.

fatman... how will you control your aiming point?

[FatMan]

Well, the IA thing sounds like a great project for our students one semester. I think we have most of the hardware around for that. For now I don't want to wait for that, I want some numbers, but who knows, I might do more of this.

As for the aim point, I don't plan to control it. I don't plan to have a bull's eye. Rather, I intend to find the centroid of the shots, and then find the deviation from that. Presumably one must adjust for range and windage when shooting anyway - the issue I'm trying to measure is consistency.

FatMan

[Redkey]

But, are you measuring the consistancy of the marker or your aim? I you don't have control over where the marker is pointed how do you know what is contributing to your spread?

A good target pellet rifle can basically put five aimed shots in one hole at 25 meters. Compare that to a pump up Crossman that can put five shots in one inch at 25 meters. If you were to randomly shoot those guns at a target without aiming you would probably get a similar spread in both shot groupings... even though you knew one gun was much better than the other.

I'd at least put a laser or scope on the gun to make sure your aim point is the same for each shot. Without controlling the aim point any spread calculations will be meaningless.

[bjjb99]

If it hasn't been stated earlier in this thread, the marker needs to be bench mounted or otherwise immobilized during the test to prevent human aim from diluting the results. In an ideal case, the marker would be solidly affixed to a firing platform and fired without anyone touching the marker itself during testing. Even something as simple as a couple of shooting sandbags upon which you can rest and steady the marker would reduce the human factor, and may well be sufficient for a test run of the experiment.

BJJB

[FatMan]

yeah, looking back at the original post I indicated I would mount the marker to a bench. I was going to use a commercial job, but instead I built one this weekend that fits my marker exactly and holds it still. I intend to strap the mount to a bench (or table, to be more precise), and then fire the marker touching only the trigger.

I didn't realize that's what you meant by controlling the aim point. I thought you meant ensuring that for each test round its pointed at exactly the same spot - there might be variation from round to round, but within a round the variation should be minimal.

Of course, I guess I could mount a laser pointer to the mount (or marker for that matter) to make sure its lined up the same, but I'm not sure that's going to buy me much.

Sorry for the confustion.

FatMan

[AGD]

Fatman,

You should also shoot each shot through a crono and decide if your going to keep the fliers or not. Another idea is to process only the shots within 3 fps and see what the deviation is compared to the whole group. This would tell you the amount of error contributed by velocity variations vs paint in flight.

AGD

[FatMan]

OK, that's a good reason to keep track of which ball made which hole. Then we can look at the whole set to get an idea of the barrel's influence on velocity consistency, and then look at groups within the same fps (+/-) and get an idea of non-velocity related consistency.

Cool! I'm getting pretty fired up about this. I need to work on my target system (based on AGD's suggestions).

I'm trying to get cphilip fired up about assisting, too.

FatMan