Teaching Pt. 14

The effectiveness of item analysis of standardized exams was demonstrated recently. At a meeting of instructors we reviewed the results of a standardized student learning outcome exam. We performed item analysis and found that most students were performing poorly on certain questions.

The purpose of the exam was to determine exactly how much the student learned and mastered by the end of the class. So if they performed poorly on certain questions the conclusion is that they must not have learned it that well.

So what's the solution? Its easy, make all your quizzes and exams comprehensive. Quite possibly the best idea of the meeting, brought up by another instructor. Given that we have limited test time and that students can only study for so many things at once, we can modify this great idea and make sure to include at least one or two questions in each quiz and exam that pertain to the questions that most students miss on the student learning outcome exam.

For example suppose most students missed questions about order of operations and linear equations, so each exam and quiz will contain at least one or two questions regarding order or operations and linear equations. We can take this further and make it part of the homework. Every homework assignment will contain questions that most students miss in the student learning outcome exam. Of course you can only give these questions on exams and homework only AFTER you covered them in the course. The idea is great, constant reinforcement of difficult concepts.

There may be some instructor resistance to accepting and implementing effective ideas from others. I always remind myself of past professors that were some of the best in their field whose instruction definitely needed improvement. This goes to show that simply because someone understands and has mastered the material, doesn't mean they know how to teach it.

So if knowing the material does not imply knowing how to teach the material, then not knowing how to teach the material does not necessarily imply that you don't know the material. Hence any suggestions that may improve one's instruction should not be taken to mean one is incompetent, or ignorant, or unintelligent. Think about it, if professors who are the best in their field often have trouble teaching, doesn't mean they stop being the best in their field.

John G.

Teaching Pt. 13

Integration of all these ideas is key to creating a robust and effective educational system. Here is one such system:

All teachers will use Gradekeeper or similar software. The benefit of using computerized grading is that software reduces the amount of work and errors incurred in calculating the student grades. In addition gradekeeper or similar software files can be uploaded to a server and the server can perform statistical analysis on each student's grades and look for negative grade trends and other factors. If negative grade trends are detected then the server alerts the instructor via email, the instructor will have a chance to reverse the trend, otherwise if it continues a school counselor will be alerted and will work with the instructor to develop strategies to reverse the grade trend.

Given that at any one time an instructor may have more than one such students, the server will be connected to the national student database, the server will suggest certain strategies to reverse the trends dependent upon profile analytics. The server will look for the most similar grade trends for past students in the most similar schools around the country, then it will look for the most effective solution. The counselor-instructor team will decide if the strategies selected will be effective for their particular situation.

Another advantage is that the server can provide contact information for the instructor or school that developed the strategy, that way they can get real time advice on how to proceed.

In addition since all grades are uploaded to a server, the server can create a webpage for each student that way both the student and the student's parents can view the grades real time. In addition, any negative grade trends will be indicated on the webpage that way the student and the parents can determine exactly where the student needs to improve.

John G.

Teaching Pt. 12

Another benefit of using scantron grading along with item analysis is that an instructor can compile the results of past classes to get a more accurate view of the strengths and weaknesses of most students and more importantly how understanding and mastery of the material develops through the year.

The instructor can determine the rate of score increase for the average student and look for correlations with specific dates, teaching methodologies, and other factors. This approach will allow the instructor to determine which factors play the biggest role in increasing the score of the students.

In addition temporal test score analysis may help in developing more effective academic calendars and will reveal which subjects require more time and which require less time and more importantly will reveal exactly HOW much time is required for each subject area. The analysis may also reveal where the students begin to lag or stagnate and gives the instructor the opportunity to discover why.

John G.

Teaching Pt. 11

Given that different schools may have different approaches, we need some method to determine whether or not an effective approach to education will have a good chance of working at a particular school. We need a method to measure how similar or how different two schools are.

Method:

In this respect we may learn something from business. The question is, how do corporations know whether or not a certain franchise will work consistently in a certain location? This may sound odd, but the reasoning is, if businesses can profile something as fickle as consumer demand to determine the best location to set up a franchise, then it should be possible to use similar methods to profile the student base and other factors to determine the most effective approach to education in a particular location.

One such method is advertised in the following website:

http://www.mappinganalytics.com/customer-profiling/customer-profiling.html

This method can be applied to every school in the US and organized into a database, then if a particular school wants to improve student outcomes, it will search the database for the highest performing school with the most similar student profile and most similar factors and copy every single detail of that school.

This method will also help instructors improve student outcomes, because instructors will have a technique by which to decide between competing teaching methodologies, e.g employ the teaching methodologies used by the highest performing schools with the most similar student profile and most similar factors.

This method will also help administrators because they will have a guide as to how to organize every aspect of their school.

Given that no two schools are exactly alike, some problems may arise, this is when the online forum is helpful. Super-instructors and super-administrators will be available to help solve problems that may arise due to profile differences. In addition the case study database will be available to help schools find solutions to common problems. In fact by combining the case study database with the profile analytics, it will be only a matter of time before all problems in education are solved.

There may be problems that no school has found a solution to yet, in this case the online forum is helpful, because everyone can offer different possible solutions. Then each possible solution is rated by instructor and administrators, as to how effective the possible solution is under actual conditions. The highest rated possible solutions are then emphasized and analyzed according to the profile analytics.

Once schools across the country are working at peak performance, for their particular location. Educational research will provide methods by which to continually improve student performance.

John G.

Teaching Pt. 10 P.S.

Another advantage of using the Scantron machine to score tests is that the instructor can use the Question Item Analysis Form after test scoring to determine which areas of the test most students are weakest. This allows the instructor to then set priorities as to which areas of the test they should focus on.

John G.

Teaching Pt. 10

I heard several reports on the radio of methods and strategies that schools around the country are trying to improve test scores in challenging environments. Two methods stand out; pay teachers according to the performance of the students along with termination if their students fail to reach the standards.

This experiment does not test theories of education, rather it tests whether incentives motivate teachers, which economics has, in some respects, already shown that it does. The only problem with this approach is that it makes no sense to terminate teachers without instructing the teachers as to the best and most effective teaching systems that work for that particular school.

Given that educational research may take time to produce effective results, a more practical strategy may be more effective in the short term.

Practical Short Term Solutions:

To give teachers the best chances of succeeding it would be best to distill every single detail of the most effective teachers into system that is easy to implement. Here is an example, find the most effective teachers in your school and copy nearly everything about them. Copy their calendar - which indicate which topics are covered and when, copy their tests, quizzes, lecture notes, copy everything they use in their classrooms to the smallest detail - in other words try to replicate the results of the most effective teachers by emulating everything they do, down to the smallest details.

Some of the problems that may arise from this approach is that different teachers have different approaches, so it would be best to perform a delphi survey of the most effective teachers to develop the most effective calendar, the most effective tests, quizzes, and homeworks, the most effective lecture notes, and any other factors that make an effective curricula - for their particular school.

This approach makes it much easier for new teachers to be effective at a challenging school.

The curricula should also detail issues that aren't covered in traditional curricula, for example which sections of the book do students have the most trouble and how to address the problem. Explanations of difficult concepts that most students will understand. How to deal with the problems of many students in one classroom, how to deal with discipline problems, how to arrange the class so that discipline problems don't even have the chance of occurring? Are there any case studies of difficult students that caused problems, along with a strategy to deal with such students? In general - What issues are common to the student base that affect learning and how do we address such issues? The approaches to these issues should be included in the curricula.

Another idea is for teachers to give students tests similar to their local standardized tests on a daily or weekly basis. This way teachers can get an objective measure of the students progress AND focus on problems BEFORE the students are tested under actual conditions. This will give the teachers time to focus on portions of the test that most students find difficult and will help them identify students that may need extra help. The teachers can use a Scantron Machine with 882-E scantron form to quickly and accurately score the practice tests.

In addition practice test scores can help the teacher gauge how effective their practices are and how closely they are replicating the practices of the most effective teachers.

Lecture Analysis:

This is good idea in concept, but some problems may arise in practice. The core problem is that it is difficult for a teacher to analyze his or her lectures, if there is no standard by which it is measured. Without such a standard or at least a method by to find areas that need improvement and practical methods to ACTUALLY address these issues, it becomes nearly impossible to improve. It is not enough to simply point out all the teachers mistakes, practical solutions should be suggested. One should also balance the negative with the positive, after pointing out a mistake, point out what the teacher did correctly. Make it clear to the teacher that the analysis is not meant to punish them, rather to help them improve.

It may be helpful if the most effective teachers review the lectures and offer their advice. They can better guide the teacher to continually improve their teaching methods.

Networking:

These are not the first nor the last schools with challenges, schools such as these have existed for a long time, yet most approaches are still trial and error. To help avoid trial and error it would helpful if the most effective, field tested practices for similar schools were shared around the country and around the world.

It would also be helpful if there was a centralized network where similar schools could exchange information. For example it may be helpful to set up an online forum where teachers around the country and around the world could exchange ideas, anonymously if they wish. One advantage of this method is that the most effective teachers around the country and around the world, for schools in similar situations, could publish their calendars, quizzes, tests, homeworks, lecture notes.

Super-teachers would be available to answer questions, solve problems, suggest methods and strategies. Super teachers would be identified according to their ratings- similar to how the rating systems work at Physicsforums.com.

To help replicate successful student outcomes, every detail of the most effective approaches, for schools in similar situations, would be published on the forum. Ineffective approaches would also be published so that schools in similar situations would AVOID making the same mistakes.

It may also be beneficial to construct a database of Case Studies of difficult students (identity protected) along with effective solutions, difficult classes along with effective solutions, concepts that students find most difficult along with effective explanations, problems with student crowd control along with effective approaches, etc. By effective I mean, field tested solutions to problems.

The most important question is, what have we learned about the best approaches for challenging schools? The practices don't have to be perfect, they simply have to take the best of what is possible in those schools.

John G.

Teaching Pt. 9

There are more issues to address in regards to education.

Scale

The problem of scale deals with the student to teacher ratio. We can best understand the problem of scale by considering certain examples.

Consider a class with only one student and one teacher, given enough time and effort, most of the time, the teacher can teach the student to perform perfectly on almost any test and to fully understand the material. Consider 2 students, it is still possible to do the same with both students, with a minimal expenditure of additional time. Consider more and more students, little by little the expenditure of additional time and effort increase until the individual approach to education ceases to be practical and in fact is counterproductive. The amount of time to help each student adds up until much of class time is spent simply helping each person.

So what must be researched is how to organize class time, class layout, class groups, to compensate for problems of scale. One idea is for the teacher to give an assessment test at the beginning of the class year to determine which students might need more help than others. The students who scored in the first quartile will be given first priority in help, the second quartile students will be given second priority, and the third quartile students will be given third priority.

In fact the third quartile students can be helpful in groupwork, they can help explain concepts to the second and first quartile students.

Students can be organized into groups of 2 or 3 where each group can contain a mix of 1,2, or 3rd quartile students or it might be more effective to group the first quartile students together in groups and mix the 2 or 3rd quartile students in groups. In any case these are the types of questions that education research can investigate - what are the optimum class organization methods to maximize student performance in large scale classrooms and with students of differing preparation? Maybe the optimum method must vary according the overall preparation of students in the class, meaning if the distribution of well prepared students is that of a bell curve then one method will be best, if the distribution of well prepared students is that of J-curve then another method must be employed. If the distribution is that of a rectangle then another method will be best.

Metrics

Metrics are an important part of educational research because, in theory, they provide objective measures of student performance. The problem is that numbers by themselves do not provide objective information, the methods by which the numbers are obtained must also be investigated.

Some examples of metrics are assessment test, class test, quiz, homework, and classwork scores. These scores can be analyzed statistically to determine if any negative trends exist, giving the teacher prior warning of problems that may arise in the near-future. This way the teacher has time to address the problems before they shows up, which is much easier than addressing the problems when they are in full effect.

John Gonzalez

Teaching Pt 8

The ideas presented in this blog are mostly directed towards schools that need improvement. A high performing or even average performing school may not need to consider all the ideas, since they have a system that works, more or less.

The scientific approach is not meant to stifle ideas, discussions, or debate. Vigorous discussions are common in all the sciences, it serves as a method by which to determine which theories are worthy to be tested and which don't work, even in the abstract. In the end, though, it is experiment that determines which theory is closer to reality. Without a test to falsify competing theories scientific progress is very difficult.

I suppose this is the ultimate form of optimism, the belief that with enough thought and enough work most problems can be solved.

One criticizes, not to personally offend, but to improve. Problems are identified and analyzed in the light of day, not to embarrass, but to solve them. Failure scenarios are considered, not to be negative, but to design systems to avoid them. Counterarguments are developed, not to bicker, but to vigorously articulate and develop one's ideas. Ideas are rigorously tested, not to stifle creativity or freedom, but to guide us towards truth. Non-falsified theories are standardized, not to dictate how to think, because it is easier to learn, master, and contribute when you have set system of thought, rather than to try to re-derive centuries of thought in one lifetime.

Field Testing

In keeping with the principle of field testing, we should welcome feedback from the teachers and the students. Just as in software testing, it is often difficult to test for every single scenario, and sometimes the theory or approach may not have taken into account certain factors that affect the student learning outcomes. Allowing for honest feedback is a great way to find weak spots in the educational systems approach. Given that teachers may be reluctant to offer honest feedback, due to administrative concerns, we may have to look at online forums where teachers are able to post anonymously.

More Later

John G.

Teaching Pt 7

The most important part of this new model of education is field testing. Even if people dismiss all the other ideas, the one idea that cannot be dismissed is does it work with actual students, in actual schools, with actual teachers, under actual conditions. No matter how well argued or well conceived a theory may be, if it doesn't work under actual conditions, then it really doesn't improve education.

The main problem is that philosophies, theories, conjectures, principles, guesses, ideas, approaches, strategies, tactics, and methods can all be argued ad infinitum. Almost any approach to education can be made to make sense in an ideal world, but in the actual world some approaches work better than others. The reason why science has progressed and given applicable results in the last 500 years is because the ultimate arbiter of theory is experiment, more importantly experiments that falsify theories.

One problem is that theories are often tested with ideal students, in ideal schools, with ideal teachers, under ideal conditions. Though this may be a good preliminary method to develop and test the theory, it is not rigorous enough to determine if it works at the level of the local school. Certainly not rigorous enough to determine if it works at the level of a challenging school. Hence before an approach to education is standardized it has to be rigorously tested at different types of schools.

This is not to say that theories are unimportant, for in order for science to progress new theories must be developed, but the theory in and of itself is not enough. Another reason why the scientific method has worked so well is because no one has to take the word of scientific authority on its face, anyone with enough equipment can test the theory on their own and determine if the theory works for themselves. Teachers don't have to believe the theory on word of the experimenters. Every teacher can test the theory in their own classrooms and determine for themselves if the theory works as it should.

Any approach to education has to be broad enough to meet the educational needs of the target student base, easy enough so that the average teacher can master and implement it, flexible enough to deal with different conditions, resilient enough to come back from difficulties, and finally must always be seen as a work in progress.

John G.